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ABSTRACT 


A  dramatic  increase  in  the  past  three  years  in  the 
Indonesian  defense  budget,  from  approximately  US$800 
million  (FY2002$)  in  2002  to  approximately  US$  2.4  billion 
(FY2005$)  in  2005,  has  created  the  need  for  the  Indonesian 
Department  of  Defense  (DOD)  and  National  Armed  Forces  (TNI) 
to  enhance  performance  and  accountability  for  effective  and 
efficient  use  of  state  funding.  It  is  imperative  that  DOD 
and  TNI  move  to  better  inform  the  public  and  high  level 
government  officials  by  increasing  transparency  on  how  the 
defense  budget  is  allocated  and  spent.  Current  DOD  and  TNI 
performance  evaluation  tools  and  financial  management 
measurement  and  reporting  methodologies  are  ineffective, 
particularly  because  they  do  not  link  performance 
assessment  to  the  budget  and  the  budget  process. 

Reform  is  needed  to  implement  Performance  Based 
Budgeting  (PBB)  to  create  direct  linkages  between 
allocation  of  resources  in  the  budget,  performance 
measurement,  and  strategic  planning  relative  to  defense 
policy  objectives  so  as  to  bolster  public  accountability 
for  the  DOD  and  TNI.  This  thesis  analyzes  implementation  of 
the  Government  Performance  Results  and  Act  of  1993  in  the 
US  Federal  Government  and  specifically  in  the  US  Department 
of  Defense,  to  develop  a  PBB  model  for  the  Indonesian 
DOD/TNI.  The  thesis  addresses  critical  factors  and  the 
steps  and  processes  necessary  to  create  a  performance-based 
budget  including  strategic  planning,  performance 
measurement,  and  methods  to  link  performance  to  budgets. 
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EXECUTIVE  SUMMARY 


In  essence.  Performance  Based  Budgeting  is  a  set  of 
processes  translating  the  strategic  planning  framework  into 
performance  measures  in  terms  of  input,  output,  and  outcome 
to  be  tied  into  resource  allocation  in  the  budgeting 
process  to  seek  greater  efficiency  and  effectiveness. 

Past  initiatives  and  experience  as  well  as  support 
from  the  President  and  Congress  have  made  GPRA  a  prominent 
best  practice  to  implement  performance  based  budgeting  in 
the  US  federal  government.  Specifically  in  the  DOD,  facing 
the  complexities  of  its  gigantic  budget  and  business 
activities  plus  the  daunting  task  inherent  in  developing 
measures  of  ultimate  performance,  progressively  DOD  has 
tried  to  incorporate  links  between  budget  development  under 
0MB  guidance  and  resource  the  allocation  decision  making 
process  PPBS . 

To  be  able  to  implement  PBB  in  2006,  the  Indonesian 
Government  and  especially  DOD  has  to  better  prepare  the 
strategic  framework  and  implementation  guidance  of  PBB, 
especially  the  performance  assessment  tool  to  be  used  to 
link  performance  to  budgets.  There  are  two  key  challenges 
for  the  Indonesian  DOD  and  TNI  to  succeed  in  implementing 
PBB.  First,  DOD  and  TNI  must  provide  a  unitary  strategic 
plan  that  requires  a  total  fusion  of  DOD  and  TNI  goals  and 
objectives.  Second,  PBB  requires  transparency.  The  degree  of 
confidential  justification  in  the  defense  sector  needs  to  be 
examined  and  redefined  so  that  transparency  in  cooperating 
with  the  stakeholders  in  designing  performance  measurement 
systems  and  linking  performance  to  budgets  will  enhance 
public  accountability  in  allocating  and  managing  limited 
budget  and  resources. 
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I .  INTRODUCTION 


A.  BACKGROUND 

In  the  past  two  decades,  there  have  been  radical 
changes  in  public  sector  management.  Although  the  issues 
between  developed  and  developing  countries  are  different, 
in  essence,  the  main  political  theme  is  how  to  bolster 
public  management  accountability.  Key  stakeholders  such  as 
the  public  and  the  legislature  have  become  increasingly 
interested  in  how  public  funds  are  allocated  and  spent  and 
how  to  reduce  public  spending  and  to  save  the  taxpayer 

money  coupled  with  attempts  to  eliminate  a  large  portion  of 
moral  hazard,  the  risk  that  the  presence  of  action  will 
affect  other  parties,  and  adverse  selection,  a  situation 
where  asymmetric  information  results  in  harming  other 
parties,  in  the  public  sector. 

One  of  the  attempts  is  to  fundamentally  restructure 
the  budget  management  system.  Shifting  from  the  previous 

view  of  highly  centralized  budget  management  focused 
primarily  on  resource  allocation  and  input  control  to  a  new 
budget  management  model,  called  "performance  budgeting," 
aimed  to  create  a  direct  linkage  between  allocating 
resources  through  the  budget  and  performance  integration  in 
reaching  stated  objectives  has  become  a  prominent  reform. 
Although  the  systems  and  implementations  vary  from  country 
to  country,  the  objective  is  solely  directed  to  provide 
public  sector  budgeting  information  on  an  outcome  basis 

seeking  greater  efficiency  and  effectiveness. 

The  idea  of  performance  budgeting  in  the  United  States 
was  initiated  in  the  1950' s  based  on  Hoover  Commission 

However,  changes  in  administration  have 
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recommendation . 


also  created  changes  in  the  budget  management  system.  From 
the  Hoover  recommendation,  to  Planning,  Programming  and 
Budgeting  System  (PPBS) ,  Management  by  Objectives  (MBO) , 
Zero  Based  Budgeting,  and  finally  Performance  Based 
Budgeting  has  enriched  the  evolution  of  using  performance 
measurement  in  budgeting  process. 

The  GPRA  is  the  recent  attempt  of  introducing 
performance-based  management  and  budgeting  techniques  in 
the  United  States.  The  Recent  economic  downturn  coupled 
with  fiscal  stress  caused  by  September  11  attacks  and  its 
further  effort  in  war  of  terrorism  has  created  significant 
budget  pressures  not  only  in  federal  government  but  also 
state  government  as  well  as  an  increasing  demand  from  the 
taxpayer  for  government  efficiency  and  effectiveness.  This 
emphasizes  the  need  for  reform.  The  information  about  the 
performance  of  programs  has  never  been  more  important. 

The  role  of  United  States  Department  of  Defense  in 
performance  budgeting  evolution  is  very  important.  PPBS 
which  was  initiated  by  Robert  McNamara,  former  Secretary  of 
Defense,  has  provided  an  introduction  of  a  decision-making 
framework  to  the  executive  branch  budget  formulation 
process  by  presenting  and  analyzing  choices  among  long-term 
policy  objectives  and  alternative  ways  of  achieving  them  as 
well  as  a  basis  in  developing  performance  plans  and 
measurement  system. 

Indonesia,  as  a  developing  country  and  fledging 
country  in  democracy,  since  a  tragic  monetary  and  economic 
crisis  in  1998,  has  experienced  a  number  of  reform  actions 
towards  how  government  can  run  like  a  business.  The  country 
has  faced  remarkable  challenges:  economic  reform,  a 
transition  to  democracy  after  30  years  under  military  rule. 
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state  owned  enterprises  restructuring  and  privatization, 
rising  unemployment  and  poverty  and  a  massive 
decentralization  of  power  to  nearly  400  local  governments. 
The  country  still  continues  to  suffer  due  to  political  and 
economic  uncertainty.  Other  public  accountability  problems 
such  as  waste  and  inefficiencies,  overseas  loan  management, 
and  endemic  corruption  have  really  caught  public  attention 
and  creditor  interests. 

The  effort  to  use  performance  budgeting  in  state 
government  is  still  in  process  in  setting  up  the 
performance  measurement  system  and  preparing  the  regulation 
to  support  the  program.  Although  in  some  local  governments 
performance  budgeting  has  been  initiated  by  having 
assistance  from  some  Non  Governmental  Organizations  (NGO) , 
the  standardized  benchmarking  performance  measurement 
system  is  still  not  available.  Past  initiatives  to 
implement  a  government  performance  evaluation  tool  had 
always  failed  because  the  government  itself  was  not  serious 
or  even  some  feel  reluctant  to  implement  it. 

In  the  defense  sector,  a  series  of  reform  agenda  has 
been  undertaken  to  alter  Indonesian  National  Armed  Forces, 
TNI  (Tentara  Nasional  Indonesia) ,  and  dominant  role  during 
the  years  of  Soeharto' s  New  Order  regime  into  a  new 
paradigm  of  military  reform.  Challenges  such  as  controversy 
on  division  of  authority  between  TNI  and  Department  of 
Defense,  military  business  under  TNI  Foundation  funds, 
minimum  defense  budget  allocation  coupled  with  lack  of 
deployment  readiness  due  to  inappropriate  and  obsolete  main 
weaponry  system,  and  inefficiency  and  ineffectiveness  in 
resource  allocation  are  still  major  concerns  for  the 
future . 


3 


DOD/TNI  has  already  implemented  the  PPBS  system  in 
which  initial  efforts  to  measure  performance  had  already 
begun.  However,  my  organization.  The  Audit  Board  of  the 
Republic  of  Indonesia  faced  difficulties  in  obtaining  and 
evaluating  information  because  there  is  no  standard  or 
benchmarking  to  be  compared  with  and  there  is  no  law  that 
forces  them  to  disclose  it  to  the  Audit  Board. 
Confidential  justification  to  protect  national  security  is 
always  used  to  cover  many  issues  in  some  areas  of  the 
military  sector  to  avoid  scrutiny  from  the  stakeholders. 

As  a  result,  the  Audit  Board,  as  an  independent  high 
state  institution  and  government's  accountability  watchdog, 
will  face  enormous  challenges  in  its  function  as  an 
external  control  towards  government  accountability  in 
utilizing  and  managing  the  state's  finances  to  create  good 
governance.  The  task  is  even  much  more  difficult  with  the 
Law  No.  7/2003  on  State  Budget  because  the  law  mandates  the 
Audit  Board  to  conduct  audits  on  government  performance.  On 
the  other  hand,  the  current  government  performance 
evaluation  tool  still  can  not  represent  an  ideal 
performance  measurement  system  due  to  lack  of  strategic 
planning  and  inappropriate  performance  measures. 

Based  on  above  circumstances,  new  tools  to  enhance 
government  accountability  and  to  cure  all  endemic  and 
systematic  government  problems  have  to  be  found.  Hopefully, 
Performance  Based  Budgeting  will  be  one  of  them. 

B.  OBJECTIVES 

The  objective  of  this  thesis  is  to  investigate  and 
analyze  the  implementation  of  Performance  Based  Budgeting 
(PBB)  in  US  Federal  Government  especially  in  DOD  as  well  as 
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to  provide  and  promulgate  a  PBB  model  for  Indonesian 
DOD/TNI.  In  addition,  this  research  will  address  the 
historical  perspective  of  PBB;  steps  and  processes  to 
implement  PBB;  the  current  implementation  Government 
Performance  and  Results  Act  (GPRA)  of  1993  including 
policy,  guidance,  tools  and  lessons  learned; 
Recommendations  will  include  why  the  PBB  model  is  important 
for  Indonesian  DOD/TNI  and  how  PBB  might  be  implemented 
throughout  the  Indonesian  DOD/TNI.  To  conclude,  this  study 
will  provide  conclusions  and  recommendations  that  could 
promulgate  the  implementation  of  PBB  in  all  branches  of  the 
Indonesian  government  especially  in  DOD/TNI. 

C .  RESEARCH  QUESTIONS 

The  research  questions  that  will  be  investigated  and 
analyzed  in  this  project  report  are  as  following: 

•  How  has  Performance  Based  Budgeting  evolved 
historically  in  United  States? 

•  What  are  GPRA  elements,  budget  and  performance 
integration  and  Program  Assessment  Rating  Tool? 

•  How  is  the  implementation  of  GPRA  and  specifically 
at  Department  of  Defense? 

•  How  does  the  US  experience  differ  from  other 
countries  (Britain,  Canada,  Australia) ? 

•  What  is  Performance  Based  Budgeting  and  what  are  the 
principles  in  PBB? 

•  What  are  steps  and  processes  that  must  be  done  in 
implementing  Performance  Based  Budgeting? 

•  How  can  performance  measures  be  linked  to  budget? 
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•  How  is  the  current  budgeting  process  in  Indonesia 
and  specifically  at  DOD/TNI? 

•  How  might  Performance  Based  Budgeting  be  implemented 
trough  Indonesian  DOD? 

D.  SCOPE  AND  LIMITATIONS 

This  study  provides  a  broad  overview  of  performance 
based  budgeting  especially  the  Government  Performance 
Results  and  Act  (GPRA)  1993.  The  focus  is  on  the  strategic 
framework  and  the  basic  process  of  performance  based 
budgeting  to  appropriately  provide  a  basic  model  for  PBB 
for  Indonesian  DOD/TNI.  Due  to  the  study  solely  based  on 
the  literature  review,  it  does  not  address  any  detailed 
implementation  of  GPRA  in  the  federal  government  and  U.S. 
DOD  as  well  as  any  detailed  model  including  evaluation  for 
Indonesian  DOD/TNI.  It  simply  provides  the  foundation  for 
further  action  or  research  into  the  potential  creation  of 
such  model  as  the  performance  based  budgeting  in  Indonesia 
concept  is  still  in  the  developmental  stages. 

E .  RESEARCH  METHODOLOGY 

The  methodological  approach  used  in  this  MBA  Project 
consists  of  conducting  a  literature  review  of  Performance 
Based  Budgeting  theory,  best  practice,  policy  and 
guidelines,  use  of  library  information  resources,  and 
websites.  It  evolved  as  the  research  project  and  process 
for  a  period  of  approximately  nine  months.  Initial  stages 
were  begun  by  performing  an  extensive  literature  review  of 
the  performance  based  budgeting  in  United  States  as  well  as 
the  GPRA.  It  was  followed  by  examining  the  conditions  in 
Indonesia  which  includes  the  budgeting  process  and  the 

6 


performance  budgeting  efforts.  After  deriving  a  number  or 
lessons  from  those  reviews,  the  focus  shifted  to  a 
comprehensive  review  of  performance  measurements  system  by 
examining  performance  measurement  theory,  policy  and 
guidelines,  and  best  practice  or  worldwide  benchmarking. 

F.  ORGANIZATION  OF  STUDY 

The  study  is  organized  into  five  chapters.  Chapter  I 
provides  the  research  background  as  well  as  delineates  the 
objectives,  research  questions,  scope  and  limitations  and 
methodologies  of  the  study. 

Chapter  II  describes  PBB  evolution  in  U.S.,  discuss 
GPRA  including  its  elements,  budget  and  performance 
integration,  general  implementation  in  federal  government 
and  specific  implementation  in  DOD  and  finally  provides 
some  comparative  studies  in  other  countries,  Britain, 
Canada,  and  Australia. 

Chapter  III  introduces  the  performance  based  budgeting 
and  describes  the  literature  review  of  Performance  Based 
Budgeting  including  strategic  planning,  performance 
measures,  and  linking  performance  and  budget. 

Chapter  IV  discusses  the  current  condition  in 
Indonesia  including  the  budgeting  process  and  performance 
budget  efforts  with  focus  on  Indonesian  Department  of 
Defense  and  provides  model  and  an  example  for  Indonesian 
DOD  about  implementing  the  performance  based  budgeting. 

Chapter  V  provides  the  conclusions  from  the  research 
concerning  the  performance  based  budgeting  and 
recommendations  for  further  study. 
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II.  GOVERNMENT  PERFORMANCE  AND  RESULTS  ACT  1993 


A.  INTRODUCTION 

This  chapter  lays  out  the  literature  review  of  the 
Government  Performance  and  Results  Act  (GPRA)  1993.  Section 
B  describes  the  performance  budgeting  evolution  in  United 
States.  Section  C  provides  the  GPRA  overview  and  the 
followed  by  the  GPRA  main  elements  in  Section  D.  Section  E 
outlines  the  Budget  and  Performance  Integration  as  well  as 
the  balance  scorecard.  Section  F  mainly  discusses  the 
Program  Assessment  Rating  Tool  (PART)  system.  Section  G 
discusses  general  implementation  of  GPRA  in  Federal 
Government.  Section  H,  reviews  specifically  the 

implementation  of  the  GPRA  in  DOD.  Section  I  reviews  the 
comparative  review  of  three  countries,  Britain,  Canada  and 
Australia.  A  summary  is  provided  at  the  end  of  the  chapter 
in  Section  J. 

B.  PERFORMANCE  BUDGETING  EVOLUTION  IN  UNITED  STATES 

Budgeting  in  the  United  States  has  experienced  at 

least  five  emphases,  starting  with  control  at  the  turn  of 

the  century,  moving  to  management  in  the  New  Deal  and  post- 

World  War  II  period,  to  planning  in  the  1960s, 

prioritization  in  the  1970s  and  1980s  and  to  accountability 

in  the  1990s.i  GAO  observes  that: 

At  the  federal  level,  interest  in  performance 
budgeting  has  led  to  numerous  initiatives  since 
World  War  II,  including  four  that  were  government 
wide  in  scope:  (1)  reforms  flowing  from  the  first 
Hoover  Commission  in  its  efforts  to  downsize  the 


1  Charlie  Tyer  and  Jennifer  Willand,  "Public  budgeting  in  America:  A 
twentieth  century  retrospective, "  Journal  of  Public  Budgeting , 
Accounting  &  Financial  Management  9,  no.  2  (1997) :  189. 
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post-World  War  II  government,  (2)  Planning- 
Programming-Budgeting-System  (PPBS)  begun  in  1965 
by  President  Johnson,  (3)  Management  by 
Objectives  (MBO)  initiated  in  1973  by  President 
Nixon,  and  (4)  Zero-Base  Budgeting  (ZBB) 
initiated  in  1977  by  President  Carter.  Each  of 
these  efforts  established  unique  procedures  for 
linking  resources  with  results. 2 


Budget  reform  was  initiated  in  1910  by  President  Taft 


by  forming  the  Commission  on  Economy  and 
report  from  commission  entitled  The  Need 
Budget  was  presented  to  Congress  in  1912 
national  attention  on  budgeting  and 
management.  Among  major  recommendations 
following : 


Efficiency.  The 
for  a  National 
and  it  focused 
sound  fiscal 
made  were  the 


1.  The  President  should  prepare  and  present  a 
budget  to  Congress  (the  executive  idea) . 

2.  A  budget  message  should  accompany  the  budget 
and  should  outline  policy  proposals  of  the 
President  as  well  as  include  summary  financial 
information . 

3.  The  Secretary  of  the  Treasury  should  submit  a 
consolidated  financial  report  to  Congress. 

4.  Each  Agency  should  submit  to  Congress  an  annual 
financial  report. 

5.  Agencies  should  establish  and  maintain  a 
comprehensive  accounting  system. 3 


These  recommendations  became  the  foundation  for  the 
Budget  and  accounting  Act  of  1921  which  established  an 
executive  budget  at  the  national  level  in  the  United  States 
and  became  the  foundation  for  present  day  budgeting  at  the 
federal  level. 


2  United  States.  General  Accounting  Office,  Performance  budgeting  : 
past  initiatives  offer  insights  for  GPRA  implementation,  (Washington, 
D.C:  The  Office,  1997),  54. 

3  Cozzettom  Kweit  and  Kweit,  1995:  20-21;  in  Tyer  and  Willand,  Public 

budgeting  in  America:  A  twentieth  century  retrospective,  189. 
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Performance  budgeting  itself  had  been  initiated  since 
1950s  based  upon  the  1949  Hoover  Commission 

recommendations,  first,  as  " a  focus  of  attention  on  the 

ends  to  be  served  by  the  government  rather  than  on  the 
dollars  to  be  spent"^  and  second,  "...a  budget  based  on 
functions,  activities  and  projects,  called  a  'performance 
budget,'  be  adopted..."  (Hoover  Commission,  1949,  p.  8.).  The 
Commission  observed  that  if  the  federal  budget  were 
prepared  on  a  performance  basis,  focusing  attention  on  the 
amount  of  work  to  be  achieved,  and  the  cost  of  this  work. 
Congressional  action  and  executive  direction  of  the  scope 
and  magnitude  of  different  federal  activities  could  then  be 
appropriately  emphasized  and  compared  in  the  resource 
allocation  process. 5 

With  the  expansion  of  governmental  activity  during  the 
New  Deal  and  World  War  II,  attention  began  to  shift  to 
management  efficiency  and  led  to  wider  interest  in 
performance  budgeting,  particularly  at  the  national  level. 
Budget  information  for  the  national  government  would  be 
structured  in  terms  of  activities  rather  than  line  items, 
and  that  performance  measurement  is  provided  along  with 
performance  report . 6 

Consistent  with  the  Hoover  Commission' s 

recommendations,  the  Congress  enacted  the  Budget  and 


4  Seckler-Hudson,  C.  Advanced  Management.:  1953,  reprinted  in  1978.  In 
Bernard  Pitsvada  and  Felix  LoStracco,  "Performance  budgeting--the  next 
budgetary  answer.  But  what  is  the  question?"  Journal  of  Public 
Budgeting ,  Accounting  &  Financial  Management  14,  no.  1  (2002) :  53. 

5  Jerry  McCaffery  and  L.  R.  Jones,  Budgeting  and  financial  management 
in  the  federal  government,  (Greenwich,  CT :  Information  Age  Publishing, 
2001),  472. 

6  J.  Burkhead,  Government  Budgeting,  New  York:  John  Wiley  and  Sons, 
1956,  In  Tyer  and  Willand,  Public  budgeting  in  America:  A  twentieth 
century  retrospective,  189. 
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Accounting  Procedures  Act  of  1950  (BAPA) ,  which,  among 
other  things,  required  the  President  to  present  in  his 
budget  submission  to  the  Congress,  the  "functions  and 
activities"  of  the  government,  ultimately  institutionalized 
as  a  new  budget  presentation:  "obligations  by  activities . 
GAO  further  states  that  these  presentations  were  intended 
to  describe  the  major  programs,  projects,  or  activities 
associated  with  each  federal  budget  request— in  a  sense,  the 
"performance  budget"  of  a  government  which  at  that  time  was 
primarily  involved  in  directly  providing  specific  goods  and 
services.  Workload  and  unit  cost  information  began  to 
appear  in  the  President's  Budget,  associated  with  the 
"obligations  by  activities"  presentations,  providing  a 
means  of  publicly  reporting  the  outputs  of  federal 
spending . 

Particularly  at  the  national  level,  however, 
performance  budgeting  encountered  a  number  of  problems  and 
performance  budgeting  did  not  work  very  well.  Budget 
estimates  were  no  more  meaningful  than  those  in  line  item 
budgets,  works  measurement  presented  problems,  measuring 
the  output  of  governmental  services  was  imprecise,  input 
could  be  easily  measured  but  not  outputs,  and,  finally, 
there  was  a  lack  of  tools  to  deal  with  the  long  range 
problems . 8 

When  Robert  McNamara  led  the  Defense  Department,  he 
initiated  a  specific  form  of  program  budgeting  called 
Program,  Planning  and  Budgeting  System  (PPBS)  in  1962  and 


'  United  States.  General  Accounting  Office,  Performance  budgeting  : 
past  initiatives  offer  insights  for  GPRA  implementation,  54. 

8  Miller,  G.J.,  Productivity  and  the  Budget  Process,"  in  Tyer  and 
Willand,  Public  budgeting  in  America:  A  twentieth  century 
retrospective,  189. 
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in  few  years.  President  Lyndon  Johnson  extended  PPPBS  to 


the  entire  federal  government.  Performance  budgeting,  as  a 
subset  of  program  budgeting,  suffered  the  same  fate  as 
program  budgeting  in  most  agencies  when  the  Nixon 
Administration  abandoned  it  in  1969.  However,  where 
remnants  of  PPBS  survived,  so  did  performance  measurements; 
in  some  respects,  the  Pentagon  led  the  recent  march  back  to 
performance  budgeting  because  PPBS  remains  its  operative 
budget  system. 9 

According  to  Allen  Schick, lO  several  other  arguments 
that  made  PBBS  failed  are: 


•  It  was  introduced  across  the  board  without  much 
preparation 

•  It  flew  in  the  face  of  existing  budgetary 
traditions  and  relationships;  in  particular, 
many  people  strongly  objected  to  the  suggestion 
that  the  budgetary  process,  which  is  inherently 
political,  could  be  made  "rational." 

•  It  was  not  given  adequate  resources,  and  top 
managers  were  not  entirely  committed  to  it. 

•  Good  analysis  and  data  were  in  short  supply, 

and  they  were  necessary  to  produce  the  kind  of 

information  crucial  to  the  success  of  PPBS. 

•  Because  PPBS  was  intended  to  require  a  review 
of  all  activities  in  each  year,  the  reform 
caused  so  much  conflict  that  the  political 
system  was  not  able  to  handle  it. 

•  PPBS  clearly  assumed  that  efficiency  was  the 

primary  value  to  be  considered  in  evaluating 
the  usefulness  of  programs,  which  was  not 
generally  agreed  to  and  is  a  difficult 

assumption  to  make  in  the  case  of  a  public 

program. 


9  United  States.  General  Accounting  Office,  Performance  budgeting  : 
past  initiatives  offer  insights  for  GPRA  implementation,  54. 

10  Schick,  Allen.  "A  Death  in  the  Bureaucracy:  The  Demise  of  Federal 
PPB."  Public  Administration  Review,  1973,  33(2),  146-156.  In  Philip  G. 
Joyce,  "Performance  Based  Budgeting,"  in  Handbook  of  Government 
Budgeting,  ed.  Roy  T.  Myers.  Anonymous  (San  Fransisco:  Josey-Bass 
Publishers,  1999),  597-619. 
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•  PPBS  was  an  executive  budget  system,  and 
largely  ignored  the  role  of  Congress  in  the 
budgetary  process.  The  result  was  that  Congress 
ignored  the  system  in  favor  of  its  established 
procedures . 

According  to  GAO  in  its  1997  report,  the  performance 
measurements  or  performance  factors  became  vital  components 
in  PPBS  analyses  and  PPBS  had  successfully  introduced  a 
decision-making  framework  to  the  executive  branch  budget 
formulation  process  by  presenting  and  analyzing  choices 
among  long-term  policy  objectives  and  alternative  ways  of 
achieving  them.  Performance  was  generally  defined  as  agency 
outputs,  with  an  agency's  program  structure  linking  outputs 
to  long-term  objectives.  Systems  analysis  and  other 
sophisticated  analytical  tools  were  an  intrinsic  part  of 
PPBS,  with  measurement  seen  as  an  essential  means  to  better 
understand  federal  outputs,  benefits,  and  costs. n 

In  1973,  President  Nixon  initiated  Management  by 
Objectives,  which  was  primarily  a  federal  management 
improvement  initiative.  GAO  observes  that  MBO  put  in  place 
a  process  to  hold  agency  managers  responsible  for  achieving 
agreed-upon  outputs  and  outcomes.  Agency  heads  were 
accountable  to  achieve  presidential  objectives  of  national 
importance;  managers  within  an  agency  would  be  held 
accountable  for  objectives  set  jointly  by  supervisors  and 
subordinates.  Performance  was  primarily  defined  as  agency 
outputs  and  processes,  but  efforts  were  also  made  to  define 
performance  as  the  results  of  federal  spending— what  would 
today  be  called  "outcomes." 


11  United  States.  General  Accounting  Office,  Performance 
past  initiatives  offer  insights  for  GPRA  implementation,  54. 
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budgeting  : 


Initiated  in  1977  by  President  Carter,  Zero-Base 

Budgeting  (ZBB) ,  an  executive  branch  budget  formulation 

process,  was  introduced  into  the  federal  government.  Zero 

Based  Budgeting,  a  bottom-up  concept  in  which  all  programs 

have  no  funding  at  the  beginning  of  each  budget  cycle, 

required  all  departments  to  identify  their  programs,  and  to 

rank  and  justify  the  existence  of  each  program  in  order  of 

priority.  GAO  claims  that: 

ZBB  main  focus  was  on  optimizing  accomplishments 
available  at  alternative  budgetary  levels  and 
under  ZBB  agencies  were  expected  to  set 
priorities  based  on  the  program  results  that 
could  be  achieved  at  alternative  spending  levels, 
one  of  which  was  to  be  below  current  funding.  In 
developing  budget  proposals,  these  alternatives 
were  to  be  ranked  against  each  other  sequentially 
from  the  lowest  level  organizations  up  through 
the  department  and  without  reference  to  a  past 
budgetary  base.  In  concept,  ZBB  sought  a  clear 
and  precise  link  between  budgetary  resources  and 
program  results.  12 


Related  to  the  idea  of  Hoover  Commission,  PPBS,  MBO 
and  ZBB,  according  to  GAQis,  past  initiatives,  although 
generally  perceived  as  having  fallen  far  short  of  stated 
goals,  contributed  to  the  evolution  of  performance-based 
measurement  and  budgeting  in  the  federal  government,  and 
many  concepts  introduced  by  these  initiatives  became 
absorbed  in  the  federal  government  and  persisted  long  after 
their  origins  in  PPBS,  MBO,  or  ZBB  had  been  forgotten.  For 
example,  first,  the  "obligations  by  activities" 
presentations  established  in  response  to  the  Hoover 


12  United  States.  General  Accounting  Office,  Performance  budgeting  : 
past  initiatives  offer  insights  for  GPRA  implementation,  54. 

13  ibid. 
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Commission' s  performance  budgeting  recommendations  continue 
today,  although  they  are  now  referred  to  as  "program 
activities."  Second,  PPBS  and  MBO  fostered  exploration  of 
difficult  performance  measurement  issues,  ultimately 
demonstrating  the  inherent  limitations  of  analysis  in  a 
political  environment  and  the  often  complex  and  uncertain 
relationship  between  federal  activities,  outputs,  and 
outcomes.  Finally,  ZBB  illustrated  the  usefulness  of 
presenting  alternative  funding  levels  and  expanded 
participation  of  program  managers  in  the  budget  process. 

When  viewing  collectively  the  past  initiatives,  GAO 
suggests  two  common  themes.  First,  any  effort  to  link  plans 
and  budgets  must  explicitly  involve  both  branches  of 
government,  executive  and  Congress.  The  failure  of  PPBS  and 
ZBB  was  solely  caused  by  lack  of  Congressional  involvement. 
Second,  with  all  past  initiatives  and  efforts  from  Hoover 
Commission,  PPBS,  and  ZBB,  it  is  more  likely  that  the 
concept  of  performance  continues  to  evolve, 

During  the  Reagan  presidency,  major  adds  to  defense 
spending  compared  to  cuts  on  non-defense  spending,  led  to 
enormous  increasing  spending  on  discretionary  accounts  that 
caused  incremental  reductions  to  federal  government  agency 
programs  and  substantial  growth  in  the  federal  budget.  By 
the  end  of  1990s,  the  country  condition  was  even  worst  with 
dollars  overflowing  the  Treasury,  the  caps  could  not 
restrain  political  will  and  the  budget  continued  to  expand. 


United  States.  General  Accounting  Office,  Performance 
past  initiatives  offer  insights  for  GPRA  implementation,  54. 
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budgeting  : 


All  of  this  leads  to  think  about  implementing  performance 
budgeting. 

In  the  1990s,  considerable  attention  has  been 
refocused  upon  performance  budgeting,  resulting  in  a  new 
notion  of  performance  budgeting  to  what  called 
"Entrepreneurial  Budgeting. 16  one  influence  was  the  1992 
publication  of  "Reinventing  Government,  How  the 
Entrepreneurial  Spirit  is  Transforming  the  Public  Sector" 
by  David  Osborne  and  Ted  Gaebler.  The  idea  was  to  hold 
governments  accountable  for  results  rather  than  focus  upon 
inputs  as  traditional  budgets  and  management  did.  Cost 
saving  and  entrepreneurial  spirit  would  be  rewarded.  A  long 
term  view  would  be  facilitated  in  terms  of  strategy,  costs 
and  planning  for  programs . 

Drawing  on  this  attention,  the  Government  Performance 
and  Results  Act  (GPRA)  of  1993  (PL.  103-62)  was  passed  by 
Congress.  The  GPRA  drew  upon  earlier  antecedents,  such  as  a 
report  submitted  during  the  Reagan  Administration  and  a 
bill  introduced  in  1991  by  Senator  William  Roth  that  had 
not  been  enacted,  but  was  revised  and  became  the  GPRA.  is 
Initially,  the  act  specifically  focused  attention  on 
results  and  performance  budgeting.  It  requires  federal 
agencies  to  prepare  strategic  plans  by  1997,  to  prepare 
annual  performance  plans  starting  with  fiscal  year  1999, 


15  H.  Heclo,  Executive  Budget  Making.  In  Pitsvada  and  LoStracco, 
Performance  budgeting--the  next  budgetary  answer.  But  what  is  the 
question?,  53. 

16  D.A.  Cothran,  Entrepreneurial  Budgeting:  An  Emerging  Reform,  (1993) 
in  Miller,  G.J.,  Productivity  and  the  Budget  Process,"  in  Tyer  and 
Willand,  Public  budgeting  in  America:  A  twentieth  century 
retrospective,  189. 

17  ibid. 

18  w.  Groszyk,  Implementation  of  the  Government  Performance  and  Results 
Act  of  1993,  1995,  November,  in  ibid. 
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and  submit  an  annual  program  performance  report  to  the 
President  and  Congress  comparing  actual  performance  with 
their  plans  beginning  in  the  year  2000. 

Basically,  the  new  performance  budgeting  has  been 
drawn  from  previous  performance  budgeting  ideas  and  program 
budgeting/PPBS  concept  as  well.  The  mechanism  for  the 
transformation  of  budgeting  was  a  blend  of  rationality  and 
Reaganism.19  Also,  like  the  previous  reforms,  performance 
budgeting  was  built  on  assumptions  about  the  ability  of  a 
change  in  process  to  change  outcomes,  without  understanding 
the  mechanism  by  which  that  would  be  accomplished  in  a 
political  system. 20 

An  important  distinguishing  aspect  of  GPRA  is  that  it 
is  a  law;  the  others,  PPBS  and  ZBB,  were  established 
through  Presidential  directive.  Therefore,  GPRA  is  created 


by  both  President  and 

Congress  and. 

It 

is  expected  that 

GPRA  would  not 

repeat 

the 

failure 

its 

predecessor.  To 

maintain  the 

efforts , 

0MB 

needs 

to 

be  more  highly 

cooperative  not 

only  with 

agencies 

but 

also  with  other 

stakeholders,  GAO  and  more  importantly  the  Congress. 

C .  GPRA  OVERVIEW 

For  decades,  there  has  been  a  perception  from  the 
public  that  American  government  institutions  and  Congress 
are  not  working  well  and  held  in  low  esteem  by  the  American 
People.  The  Committee  on  Government  Affairs  believes  that 
part  of  the  explanation  for  this  apparent  inconsistency 


19  Janet  M.  Kelly,  "The  long  view:  Lasting  (and  fleeting)  reforms  in 
public  budgeting  in  the  twentieth  century, "  Journal  of  Public 
Budgeting ,  Accounting  &  Financial  Management  15,  no.  2  (2003)  :  309. 

20  R.C.  Moe,  The  "reinventing  government"  exercise:  misinterpreting  the 
problem,  misjudging  the  consequences.  Public  Administration  Review, 
54(2),  111-122,  1994.  In  ibid. 
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could  be  seen  from  a  public  opinion  poll  which  showed  that 
Americans,  on  average,  believe  that  as  much  as  48  cents  out 
of  every  Federal  tax  dollar  is  wasted.  In  other  words,  the 
public  believes  that  it  is  not  getting  the  level  and 
quality  of  government  service  for  which  it  is  paying. 21 

The  committee  also  realizes  that,  first,  waste  and 
inefficiency  in  Federal  programs  undermine  the  confidence 
of  the  American  people  in  the  Government  and  reduce  the 
Federal  Government's  ability  to  adequately  address  vital 
public  needs.  Second,  Federal  managers  are  seriously 
disadvantaged  in  their  efforts  to  improve  program 
efficiency  and  effectiveness,  because  of  insufficient 
articulation  of  program  goals  and  inadequate  information  on 
program  performance.  Finally,  congressional  policymaking, 
spending  decisions  and  program  oversight  are  seriously 
handicapped  by  insufficient  attention  to  program 
performance  and  results. 

The  purposes  of  GPRA  are  to-(l)  improve  the  confidence 
of  the  American  people  in  the  capability  of  the  Federal 
Government,  by  systematically  holding  Federal  agencies 
accountable  for  achieving  program  results;  (2)  initiate 
program  performance  reform  with  a  series  of  pilot  projects 
in  setting  program  goals,  measuring  program  performance 
against  those  goals,  and  reporting  publicly  on  their 
progress;  (3)  improve  Federal  program  effectiveness  and 
public  accountability  by  promoting  a  new  focus  on  results, 
service  quality,  and  customer  satisfaction;  (4)  help 
Federal  managers  improve  service  delivery,  by  requiring 
that  they  plan  for  meeting  program  objectives  and  by 


21  Committee  on  Governmental  Affairs  United  States  Senate,  Government 
Performance  and  Resultas  Act  of  1993  Report , Anonymous  ,  1993),  1-55. 
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providing  them  with  information  about  program  results  and 
service  quality;  (5)  improve  congressional  decision-making 
by  providing  more  objective  information  on  achieving 
statutory  objectives,  and  on  the  relative  effectiveness  and 
efficiency  of  Federal  programs  and  spending,  and  (6) 
improve  internal  management  of  the  Federal  Government.  GPRA 
is  a  potentially  far  reaching  piece  of  legislation  and  it 
seeks  to  make  government  agencies  more  effective  and 
accountable  by  enhancing  their  strategic  planning, 
establishing  measurable  performance  objectives,  and 
eventually  tying  agency  budgets  to  the  accomplishment  of 

those  objectives22 

The  concept  draws  on  (indeed,  transplants)  concepts 
from  private  sector  management  models  and  establishes  a 
system  in  which  market-like  discipline  could  be  used  to 
improve  federal  government  management .  23  As  noted  in 
President  Bush's  2003  budget  submittal,  the  ultimate 
objective  is  to  move  the  budget  debate  from  "what  will  the 
federal  government  spend?"  to  "what  will  the  federal 
government  achieve?"24  The  Bush  administration  has  continued 
and  expanded  the  efforts  of  the  Clinton  administration  to 
integrate  performance  information  in  the  federal  budget 
process  and  to  increase  managerial  flexibility.  The 
performance  information  from  the  GPRA  process  is  to  be  used 

22  Manuel  R.  Gomez,  "Exposure  Surveillance  tools  needed  in  agency  GPRA 
plans,"  American  Industrial  Hygiene  Association  Journal  59,  no.  6 
(1998)  :  371 . 

23  Beryl  A.  Radin,  "The  Government  Performance  and  Results  Act  (GPRA) : 

Hydra-Headed  Monster  or  Flexible  Management  Tool?"  Public 
administration  review  58,  no.  4  (1998) :  307-316. 

24  Office  of  Management  and  Budget,  Budget  of  the  United  States 

Government ,  Fiscal  Year  2003  (Washington,  DC,  2002)  :  47-52.  in  R.  M. 

McNab  and  F.  Melese,  "Implementing  the  GPRA:  Examining  the  Prospects 
for  Performance  Budgeting  in  the  Federal  Government,  "  Public  Budgeting 
and  Finance  23,  no.  2  (2003)  :  73-95. 
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to  score  agencies  on  their  performance,  allocate  (and 
reallocate)  funds  among  competing  programs,  and  consolidate 
and  terminate  unnecessary  programs. 25 

According  to  Wildavsky  and  Caiden26,  there  are  three 
key  challenges  for  the  GPRA  to  be  considered  a  success. 
First,  the  GPRA  should  assist  in  the  creation  of  an 
institutional  framework  conducive  to  forming  consensus  on  a 
unique  set  of  objectives  among  conflicting  stakeholders. 
Second,  GPRA  implementation  must  support  the  ultimate 
stated  objective  of  linking  resources  to  results-or  to 
relate  data  on  program  performance  to  appropriation  account 
structures-f or  the  conjectured  efficiency  gains  to  be 
realized.  Finally,  the  GPRA  must  overcome  a  traditional 
system  of  budgeting  that,  while  often  criticized  and  the 
focus  of  almost  continuous  reform  efforts,  has  survived  to 
this  day. 

D.  GPRA  MAIN  ELEMENTS 

The  conceptual  entirety  of  GPRA  is  fused  from  three 
basic  elements:  (1)  strategic  planning,  (2)  performance 
measurement  (which  covers  goal  setting,  measurement,  and 
reporting  and  (3)  greater  flexibility  for  managers  in 
return  for  greater  accountability  for  results. 27 

A  strategic  plan  of  program  activities  covers  at  least 
a  five  year  period  and  is  updated  at  least  every  three 
years.  The  strategic  plan  contains  a  comprehensive  mission 

25  Office  of  Management  and  Budget,  Budget  of  the  United  States 
Government ,  Fiscal  Year  2003  (Washington,  DC,  2002)  :  47-52.  in  ibid. 

26  Aaron  Wildavsky  and  Caiden,  The  New  Politics  of  the  Budgetary 
Process  (New  York:  Harper  Collins,  1992)  in  ibid. 

22  Groszyk,  Implementation  of  the  Government  Performance  and  Results 
Act  of  1993, 
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statement  for  the  agency,  a  specification  of  general  goals 
and  objectives  (including  outcome-related  goals  and 
objectives)  as  well  as  how  they  will  be  achieved,  the 
relationship  between  the  general  goals  and  objectives  and 
performance  goals,  identification  of  key  factors  that  could 
affect  the  achievement  of  the  goals  and  objectives,  and  a 
description  of  the  program  evaluations  used  in  establishing 
or  revising  the  goals  and  objectives.  Each  agency  has  to 
present  the  plan  to  Congress  and  others  interested  in  or 
affected  by  the  plan  (the  stakeholders) . 

The  strategic  plan  must  contain  a  description  of  how 
the  agency  intends  to  achieve  the  general  goals  and  should 
cover  the  overall  approach  that  will  be  taken  over  the  time 
period  covered  by  the  plan,  including  a  schedule  for 
significant  actions  and  the  needed  resources.  It  should 
indicate  how  the  goals  of  the  annual  performance  plans  will 
be  used  to  measure  progress  in  achieving  the  general  goals 
of  the  strategic  plan,  and  the  underlying  basis  for  any 
assumptions  or  pro j ections .  28 

The  annual  performance  plan  includes  not  only  an 
agency's  annual  performance  plan  but  also  the  Federal 
Government  performance  plan  for  the  overall  budget.  Whereas 
the  agency  plans  are  submitted  to  0MB  for  use  in  preparing 
the  overall  government-wide  performance  plan,  the  Federal 
Government  plan  is  to  be  submitted  to  Congress  as  part  of 
the  Budget  of  the  United  States. 

The  Annual  performance  plans  provide  the  direct 
linkage  between  an  agency's  longer-term  goals  defined  in 


2o  Committee  on  Governmental  Affairs  United  States  Senate,  Government 
Performance  and  Resultas  Act  of  1993  Report,  1-55. 

22 


the  strategic  plan  and  the  daily  basis  activities.  The 
plans  include  performance  goals  that  define  the  level  of 
performance  to  be  achieved  by  a  program  activity  in 
quantifiable,  and  measurable  form;  indicate  the  operational 
process,  skills,  technology,  and  resources  required; 
provide  performance  indicators  to  be  used  in  measuring 
relevant  outputs;  establish  a  basis  for  comparing  actual 
program  results  with  the  established  performance  goals;  and 
specify  the  means  to  be  used  to  verify  and  validate 
measured  values.  29 

Annual  program  performance  reports  contain  two  main 
parts:  a  report  on  the  actual  performance  achieved  compared 
to  the  performance  goals  expressed  in  the  performance  goals 
plan;  and  of  the  steps  to  be  taken  to  achieve  those  goals 
that  were  not  met.  The  reports  provide  comparison  of  the 
actual  performance  of  each  program  activity  to  the 
performance  goals  in  the  Annual  Performance  Plan  for  a 
fiscal  year.  In  this  report,  the  agencies  have  to  explain 
what  goals  have  or  have  not  been  achieved,  why  certain 
performance  goals  have  not  been  met  in  how  to  provide  plans 
for  achieving  the  goals  in  the  future  or  to  modify  the  goal 
if  it  is  not  appropriate. 

The  agency's  performance  report  must  be  submitted  to 
the  appropriate  authorization  and  appropriations  committees 
of  the  Congress  and  upon  request,  the  Committee  can  ask 
more  specific  information  on  the  actual  performance  for  any 
performance  indicator  established  in  its  annual  performance 
plan.  Agencies  are  required  to  begin  reporting  performance 
trends,  on  a  phased-in  basis,  so  that  for  FY  2002  and 


29  Radin,  The  Government  Performance  and  Results  Act  (GPRA) :  Hydra- 
Headed  Monster  or  Flexible  Management  Tool?,  307-316. 
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thereafter,  performance  data  will  be  shown  for  each  of  the 
most  recent  four  years. 30  The  annual  performance  reports 
provide  feedback  to  managers,  policymakers,  and  the  public 
about  the  accomplishment  of  the  resources  expended  or  the 
original  goals.  The  reports  must  correlate  between 
performance  measurement  information  and  program  evaluation 
findings  to  provide  a  clear  picture  of  the  agency's 
performance  and  its  efforts  at  improvement. 

In  the  managerial  flexibility  section,  0MB  had  to 
select  at  least  five  agencies  to  participate  in  pilots  from 
among  eligible  GPRA  performance  planning  and  reporting 
agencies  during  fiscal  years  1994  to  1996.  Agency  proposals 
were  to  identify  the  constraining  requirement  to  be  waived, 
quantify  how  relief  would  affect  performance,  and  compare 
the  anticipated  performance  improvements  with  current 
performance  levels  versus  levels  that  could  be  expected 
without  the  waiver. 3i  Performance  report  shall  also  describe 
the  use  and  assess  the  effectiveness  of  any  waiver  of 
administrative  requirements  and  controls,  and  summarize  the 
findings  of  those  program  evaluations  completed  during  the 
year  covered  by  the  report.  If  the  agency  has  prepared  a 
classified  or  non-public  annex  to  its  annual  performance 
plan,  then  those  same  items  shall  be  covered  in  a 
classified  or  non-public  annex  to  the  performance  report . 32 

Any  agency  is  not  allowed  to  have  authority  to  waive 
administrative  requirements  established  in  law.  However,  if 


30  Committee  on  Governmental  Affairs  United  States  Senate,  Government 
Performance  and  Resultas  Act  of  1993  Report,  1-55. 

31  Jonathan  D.  Breul,  "The  Government  Performance  and  Results  Act--10 
years  later,"  The  Journal  of  Government  Financial  Management  52,  no.  1 
(2003) :  58 . 

32  Committee  on  Governmental  Affairs  United  States  Senate,  Government 
Performance  and  Resultas  Act  of  1993  Report,  1-55. 
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an  agency  has  authority  under  a  law  other  than  this  Act  to 
waive  a  statutory  requirement  or  control,  it  may  do  so  and 
need  satisfy  only  the  requirements  of  that  law.  33 
Requirements  and  controls  that  may  be  waived  are  those 
imposed  across  the  government  agency  which  has  a  central 
management  role,  and  cover  areas  such  as  financial 
management,  personnel,  supply  buildings  etc.  34 

E.  BUDGET  AND  PERFORMANCE  INTEGRATION 

Budget  and  Performance  Integration  (BPI)  is  one  of  the 
most  important  of  the  reforms  on  the  President's  Management 
Agenda.  0MB  made  this  point  explicitly  when  first 
announcing  the  management  agenda  by  stating,  "Improvements 
in  the  management  of  human  capital,  competitive  sourcing, 
improved  financial  performance,  and  expanding  electronic 
government  will  matter  little  if  they  are  not  linked  to 
better  results. "35  According  to  0MB,  the  reasons  why  budget 
and  performance  integration  is  very  important  in 
President's  Management  Agenda  because  scarce  federal 
resources  are  seldom  be  allocated  with  convincing  accounts 
of  the  results  and  agency  performance  measures  are 
improperly  integrated  into  agency  budget  submissions  and 
the  management  and  operations  of  agencies. 

To  solve  the  problems  and  improve  financial 
performance,  0MB  suggested  several  reform  actions  to  the 


33  Committee  on  Governmental  Affairs  United  States  Senate,  Government 
Performance  and  Resultas  Act  of  1993  Report,  1-55. 

34  Groszyk,  Implementation  of  the  Government  Performance  and  Results 
Act  of  1993, 
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[cited  2004 ] . 


Administration  such  as  integrating  performance  review  with 
budget  decisions,  identifying  high  quality  outcome  measures 
which  accurately  monitor  the  performance  of  programs, 
integrating  more  completely  information  about  costs  and 
program  performance  in  a  single  oversight  process  which 
include  budgeting  for  the  full  cost  of  resources  where  they 
are  used,  making  budget  programs  and  activity  lines  more 
parallel  with  outputs,  and  where  useful,  improving 
alignment  of  budget  accounts.  36 

By,  this  initiative,  0MB  expects  some  expected  results 
to  be  achieved  so  that  federal  managers  would  have  better 
program  performance  and  better  cost  control  and 
accountability.  First,  0MB  expects  agencies  to  implement 
performance  budgeting,  in  which  each  agency  and  program 
funding  to  be  based  on  tangible  results  and  to  use 
comparisons  between  various  programs  to  make  such  budget 
decisions.  Starting  in  2003,  the  President's  Budget  will 
shift  budgetary  resources  among  programs  devoted  to  similar 
goals  to  emphasize  those  that  are  more  effective  and  the 
Administration  will  set  performance  targets  for  selected 
programs  along  with  funding  levels. 37 

Second,  0MB  expects  that  by  early  2004,  agencies  will 
fully  understand  and  facilitate  the  Program  Assessment 
Rating  Tool  (PART)  to  obtain  an  immediate  impact  on  0MB' s 
decisions  on  program  funding  levels.  Finally,  0MB  expects 
agencies  to  have  the  necessary  financial  management  and 
accounting  systems.  The  principles  of  managerial  cost 

36  "Budget  and  Performance  Integration:  Expected  Results,"  [cited 
2004],  Available  from  http://www.john-mercer.com/bpi  results.htm 
3/  ibid. 
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accounting,  and  especially  Activity  Based  Costing  and  Cost 
Management  (ABC/M) ,  are  key  enablers  for  the  agency  to 
realize  the  full  benefits  of  performance-based  budgeting 
through  improved  day-to-day  program  management.  0MB 
mentions  the  following  as  being  among  the  expected  long¬ 
term  benefits  of  this  reform: 

o  "Better  performance..." 

o  "Better  control  over  resources  used  and 
accountability  for  results  by  program 
managers..." 

o  "Better  service..." 

o  "Standard,  integrated  budgeting,  performance, 
and  accounting  information  systems  at  the 
program  level  that  would  provide  timely 
feedback  for  management  and  could  be  uploaded 
and  consolidated  at  the  agency  and  government 
levels.  This  would  facilitate  the  goals  of  the 
President's  initiative  to  improve  financial 
performance . "38 

To  provide  a  consistent  approach  to  rating  programs 
across  the  Federal  government,  0MB  suggested  to  use  an 
assessment  took  called  The  Program  Assessment  Rating  Tool 
(PART) .  "The  Program  Assessment  Rating  Tool  (PART)  is  a 
series  of  questions  designed  to  provide.  The  PART  is  a 
diagnostic  tool  that  relies  on  objective  data  to  inform 
evidence-based  judgments  and  to  assess  and  evaluate 
programs  across  a  wide  range  of  issues  related  to 
performance . "39  As  a  central  to  the  budget  and  performance 
integration  initiative,  the  PART  is  meant  to  strengthen  the 
process  for  assessing  the  effectiveness  of  programs  by 
making  that  process  more  robust,  transparent,  and 


3o  "Budget  and  Performance  Integration:  Expected  Results,"  [cited 
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systematic.  The  PART  covers  seven  categories  of  Federal 
Programs  are:  Direct  Federal  Programs,  Competitive  Grant 
Programs,  Block/Formula  Grant  Programs,  Regulatory  Based 
Programs,  Capital  Assets  and  Service  Acquisition  Programs, 
Credit  Programs,  and  Research  and  Development. 

According  to  0MB  in  its  fiscal  year  2006  PART 
Instruction, 40  the  PART  strengthens  and  reinforces 
performance  measurement  under  GPRA  by  encouraging  careful 
development  of  performance  measures  according  to  the 
outcome-oriented  standards  of  the  law  and  by  requiring  that 
agency  goals  be  appropriately  ambitious.  Therefore, 
performance  measures  included  in  GPRA  plans  and  reports  and 
those  developed  or  revised  through  the  PART  process  must  be 
consistent.  0MB  further  states  that  the  PART  also  extends 
the  usefulness  of  performance  information  by  defining 
programs  that  comprise  activities  about  which  management 
and  budget  decisions  are  made. 

In  its  review  of  the  first  year  of  PART,  by  performing 
a  simple  regression  analysis,  GA04i  found  that  PART  scores 
have  a  positive  and  statistically  significant  effect  on 
recommended  levels  in  the  president's  budget,  although  great 
deal  of  variance  was  left  unexplained.  To  examine  the  effect 
of  program  size  on  the  results,  GAO  divided  all  programs 
equally  into  three  groups-small ,  medium,  and  large-  based  on 
their  fiscal  year  2003  funding  estimate.  Regression  then  was 
performed  for  discretionary  programs  in  each  group.  The 


40  Office  of  Management  and  Budget,  0MB’ s  Program  Assessment  Rating 
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results  suggest  that  the  statistically  significant  effect  of 
overall  scores  on  budget  outcomes  exists  only  for  the 
smaller  programs.  The  regression  analysis  also  shows  that 
PART  scores  have  a  positive  and  statistically  significant 
effect  on  discretionary  program  funding.  Regression  results 
for  mandatory  programs  showed-as  expected-no  relationship 
between  PART  scores  and  the  level  of  funding  in  the 
President's  Budget  Proposal.  Assessment  ratings,  however, 
can  potentially  affect  the  funding  for  discretionary 
programs  either  in  the  President's  Budget  proposal  or  in 
congressional  deliberations  on  spending  bills. 42 

F.  GPRA  IMPLEMENTATION  IN  THE  FEDERAL  GOVERNMENT 

Although  the  act  was  passed  in  1993,  the  deadline  for 
implementation  was  1997,  the  time  when  all  executive 
agencies  were  required  to  submit  a  five-year  strategic  plan 
to  0MB  and  Congress.  To  start  the  implementation  of  the 
GPRA,  the  act  mandated  a  series  of  pilot  projects  to 
provide  experience  and  lessons  for  government  agencies.  The 
pilot  projects  for  performance  measurement  were  undertaken 
in  28  designations,  14  cabinet  departments  and  14 
independent  agencies.  Those  pilot  projects  were  finally 
started  in  fiscal  1998  after  being  delayed  for  at  least  a 
year  by  0MB. 

During  earlier  implementation,  there  were  massive 
critiques  of  the  adequacy  of  the  act  to  fulfill  its  goals 
and  objective  as  well  as  the  willingness  of  the  federal 
agencies  to  seriously  implement  it.  In  a  few  months  in 

42  United  States.  General  Accounting  Office,  Performance  budgeting 
[electronic  resource]  :  observations  on  the  use  of  0MB 's  program 
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1998,  the  Washington  press  (which  is  rarely  interested  in 
management  issues)  had  focused  unusual  intention  to  the 
implementation  of  GPRA,  reporting  various  pronouncements 
from  the  GAO,  0MB  and  the  Republican  leadership  in  Congress 
about  the  relative  success  or  failure  of  federal  agencies 
to  comply  with  its  initial  requirements  related  strategic 
planning .  43 

Beryl  Radin44  also  claims  that  the  directions  for  the 
implementation  of  GPRA  that  did  emerge  from  0MB  were 
variable.  She  said  that  in  some  instances,  0MB  staff  acted 
as  a  cheerleader  but  did  not  provide  clear  instructions  to 
agencies  on  what  was  expected,  while  in  other  instances, 
0MB  provided  the  sophisticated  equivalent  of  a  paint-by- 
numbers  exercise.  She  continues  by  observing  that  there  was 
very  little  attention  to  process  measures  and  the 
relationship  between  process  activities  and  output  or 
outcome  performance. 

GAO  in  its  1999  report45  observes  that  there  were  a 
variety  and  complexity  of  agency  planning  and  budget 
structures.  These  necessarily  resulted  in  a  range  of 
approaches  to  present  and  link  the  performance  information. 
Based  on  their  analysis,  GAO  concluded  that  some  agencies 
were  able  to  develop  informative  approaches  to  connect 
budgetary  resource  to  results,  while  some  were  not.  Most  of 
agencies  still  kept  their  plans  separated  with  their  budget 
submissions  and  only  6  of  the  35  plans  had  been  fully 
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integrated  into  the  agency's  budget  justification.  This  can 


be  seen  in  following  table: 


Table  1 :  Agencies  Status  on  Allocation  of  Program  Activity 

Funding  to  Performance  Goals 

Nonnumeric  characteristics 

Allocated 

Did  not 

allocate 

Total 

program 

activity 

program 

activity 

funding 

to 

funding 

to 

performance 

performance 

goals 

goals 

Number 

Percent 

Number 

Percent 

Number 

Concentration  of  spending 

Spending  in  multiple  accounts 

10 

42 

14 

58 

24 

Spending  concentrated  in  one  account 

4 

36 

7 

64 

11 

Program  activity  structure 

Not  common 

13 

39 

20 

61 

33 

Common 

1 

50 

1 

50 

2 

Budget  structure  used  in  the  performance  plan 

Proposed  change  in  structure 

3 

100 

0 

0 

3 

Used  fiscal  year  1998  structure 

11 

34 

21 

66 

32 

Integration  of  performance  plan 

Separate  from  budget  justification 

9 

31 

20 

69 

29 

Fully  integrated  with  budget  justification 

5 

83 

1 

17 

6 

Program  activities  linked  to 

strategic  goal 

3 

50 

3 

50 

6 

Strategic  objective 

3 

43 

4 

57 

7 

Performance  goal 

7 

50 

7 

50 

14 

other 

1 

33 

2 

67 

3 

None 

0 

0 

5 

100 

5 

Plan  Associated  dollars  with 

strategic  goal 

5 

71 

2 

29 

7 

Strategic  objective 

3 

50 

3 

50 

6 

Set  of  Performance  goals 

4 

100 

0 

0 

4 

Performance  goal 

0 

0 

1 

100 

1 

other 

1 

50 

1 

50 

2 

None 

1 

7 

14 

93 

15 

Relationship  of  program  activities  to 

performance  goals 

Simple  relationship 

8 

100 

0 

0 

8 

Complex  relationship 

6 

27 

16 

73 

22 

Source:  From  GAO  Report:  GAO/AIMD/GGD-99 

-67,  Appendix  I 1^6 

On  January  18,  2001,  based  on  review  on  the  results  of 
some  performance  budgeting  pilots,  in  its  report,  0MB 
reported  that  the  pilots  demonstrated  that  assuring  further 
performance  measurement  improvements  and  steadily  expanding 
the  scope  and  quality  of  performance  measures  was 
paramount,  and  that  the  existing  statute  provides 

46  United  States.  General  Accounting  Office,  Performance  Budgeting: 
Initial  Experiences  Under  The  Results  Act  in  Linking  Plans  with 
Budgets...  162173...  U.S.  GAO.  .Anonymous  (General  Accounting  Office 
April  12 : ,  1999)  . 
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sufficient  latitude  for  such  improvement .  47  in  the  same 
report,  0MB  also  concluded  that  the  pilots  raised  several 
key  challenges  about  performance  budgeting  such  as: 

•  In  many  instances,  measuring  the  effects  of 
marginal,  annual  budget  changes  on  performance 
is  not  precise  or  meaningful. 

•  While  continuing  to  change  from  an  almost  total 
reliance  on  output  measures  to  outcome 
measures,  it  will  be  much  more  difficult  to 
associate  specific  resource  levels  with  those 
outcomes,  particularly  over  short  periods  of 
time . 

•  Establishing  clear  linkages  between  funding  and 
outcomes  will  vary  by  the  nature  of  the  program 
and  the  number  of  external  factors . 

•  Delays  in  the  availability  of  performance  data, 
sometimes  caused  by  agencies  reliance  on 
nonfederal  program  partners  for  data 
collection,  will  continue  to  present 
synchronization  problems  during  the  budget 
formulation . 48 


In  maximizing  the  effort  to  support  GPRA,  in  April 
2001,  0MB  introduced  a  new  approach,  a  sweeping 
reorganization  to  "better  integrate  the  budget  analysis, 
management  review  and  policy  development  roles.  0MB  also 
merged  0MB' s  separate  budget  and  management  functions  into 
five  resource  management  offices  (RMOs) .  The  new  offices 
were  assigned  responsibility  for  not  only  formulating  and 
reviewing  the  budget  request,  but  also  assessing  program 
effectiveness,  conducting  mid  and  long  range  policy  and 


47  United  States.  General  Accounting  Office,  Results-Oriented 

Government:  GPRA  Has  Established  a  Solid  Foundatuon  for  Achieving 

Greater  Results,  (Washington  D.C.:  U.S.  General  Accounting  Office, 
2004),  270. 

48  ibid. 
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program  analysis,  and  implementing  government  wide 
management  policy  and  program  evaluation . 49 

Jonathan  D.  Breul,  a  fellow  and  member  of  the  Board  of 
Trustees  of  the  National  Academy  Public  Administration  and 
former  senior  advisor  to  the  deputy  director  for  Management 
in  0MB,  elaborated  his  review  on  GPRA  in  2003  after  its  10 
years  of  implementation.  Of  all  GPRA  features,  he  found 
that  three  features  of  GPRA  have  worked  and  four  features 
have  not  worked. 

Three  features  that  have  worked  are  bipartisan 
statutory  framework,  strategic  planning,  and  performance 
measurement.  For  bipartisan  statutory  framework,  the  GPRA 
has  been  supported  by  both  Democrat  and  Republican,  in 
which  it  was  conceived  by  Republican  senator,  passed  under 
a  Democrat  committee  chair  and  signed  by  Democrat 
President.  Even,  President  Bush  has  articulated  the  GPRA 
through  its  budget  performance  integration  as  a  key  of  his 
Management  Agenda. 

In  implementing  strategic  planning,  there  have  been 
dramatic  changes  in  federal  agencies.  Before  GPRA 
implemented,  there  was  no  single  strategic  plan  in  the 
federal  government;  nowadays  it  exists  in  every  department 
and  agency.  However,  according  to  Breul,  the  requirement  to 
update  the  plan  at  least  once  every  three  years  has  proved 
to  be  awkward  because  it  is  out  of  synch  with  the  four-year 
presidential  cycle. 

For  performance  measurement,  currently  the  federal 
agencies  have  improved  in  designing  their  performance 
information  as  useful  baselines  in  assessing  future  program 

49  Breul,  The  Government  Performance  and  Results  Act--10  years  later, 
58  . 
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Breul  observes  that  developing 


performance.  However, 
credible  information  on  program  results  remains  a  work  in 
progress,  as  agencies  struggle  not  only  to  define  their 
contribution  to  outcomes-  which  in  many  cases  are 

influenced  only  partially  by  federal  intervention  with 
funds,  regulations,  etc.-  but  also  to  assure  the 

reliability  of  their  performance  data. 

The  four  features  of  GPRA  that  have  not  worked  are 
managerial  flexibility  pilots,  performance  budgeting 
pilots,  government  wide  performance  plan  and  a  paper 
exercise.  For  managerial  flexibility,  of  61  waiver 
proposals  submitted  by  pilot  project  agencies,  0MB  did  not 
approve  any  of  them,  and  even  three  quarters  of  them  were 
simply  not  approved  for  statutory  or  other  reasons.  For 
the  remaining  proposals,  according  to  Breul,  0MB  or  other 
central  management  agencies  developed  compromises  by  using 
authorities  that  were  already  available  independent  of 
GPRA.  Breul  further  claims  several  factors  contributed  to 
the  failure  of  the  managerial  flexibility  pilots:  first 
were  changes  in  federal  management  practices  and  laws  that 
occurred  after  GPRA  was  enacted;  second  is  that  GPRA  was 
not  the  only  way  agencies  could  seek  relief.  In  addition  to 
other  pre-existing  authorities.  Vice  President  Gore's 
National  Performance  Review  had  launched  about  185 
reinvention  labs  to  provide  managers  added  operational 
authority,  quite  independent  of  GPRA.  50 

However,  the  pilot  process  did  provide  some  important 
lessons  for  government  wide  implementation.  Breul  continues 
by  observing  that:  0MB  and  the  agencies  learned  that  the 

50  Breul,  The  Government  Performance  and  Results  Act--10  years  later, 
58  . 
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burdens  and  constraints  that  confronted  managers  were  often 
self-imposed  by  the  agency  itself  or  its  parent  department. 
For  example,  the  Internal  Revenue  Services  sought  a  waiver 
from  a  requirement  that  envelopes  be  purchased  through  the 
General  Services  Administration-even  though  there  was  such 
no  requirement  by  GSA.51 

For  performance  budgeting  pilots,  the  problem  existed 

because  the  agencies  spent  nearly  all  of  their  time  and 

effort  ensuring  that  strategic  and  annual  plans  would  be 

useful  documents  and  met  the  requirements  of  the  statute 

and  less  devoting  their  time  to  the  real  performance 

budgeting  implementation.  This  condition  made  0MB  delay  the 

start  of  the  pilot  projects  by  one  year.  The  government 

wide  performance  plans  also  failed  because  few  agencies 

took  any  interest  in  the  plans  and  even  0MB  did  not  issue 

it  in  the  fiscal  years  2002  or  2003. 

Finally,  relating  to  his  statement  to  consider  GPRA  as 

a  paper  work  exercise,  Breul  claims  that: 

" . there  is  a  risk  of  GPRA  becoming  a  hollow, 

paperwork  exercise,  requiring  reporting  that  is 
unrelated  to  the  real  work  of  Congress  and  the 
agencies.  If  one  were  to  stack  up  all  the  GPRA 
documents  produced  for  Congress  last  year,  the 
pile  would  measure  over  a  yard  high.  Reporting  is 
not  timely  for  decision  makers,  and  is  too 
voluminous  for  users  to  find  useful  information 
(overly  footnoted,  etc.)  appropriators  and 
executive  branch  decision  makers  often  must  wade 
through  reams  of  paper  to  find  a  few  kernels  of 

useful  inf ormation . "52 


51  Breul,  The  Government  Performance  and  Results  Act--10  years  later, 
58  . 

52  ibid. 
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In  its  March  2004  report^s,  gaO  states  that  the  act  has 
laid  a  strong  foundation  for  a  more  results-oriented 
federal  government.  However,  GAO  found  several  challenges 
that  exist  to  effective  implementation  of  GPRA.  The 
challenges  are  inconsistencies  of  top  leadership  in  showing 
commitment  to  achieving  results,  report  mixed  results  in 
use  of  performance  information,  managers  confrontation  with 
a  range  of  human  capital  challenges,  persistent  challenges 
in  setting  outcome-oriented  goals,  measuring  performance, 
and  collecting  useful  data,  crosscutting  issues  in 
successful  GPRA  implementation  and  limited  use  of 
performance  information  by  Congress. 

G.  DOD  IMPLEMENTATION  OF  GPRA 

Defense  operations  involves  over  $1  trillion  in 
assets,  budget  authority  of  about  $310  billion  annually, 
and  about  3  million  military  and  civilian  employees  and 
these  operations  represents  one  of  the  largest  management 
challenges  within  the  federal  government.  with  these 

complexities,  the  development  of  appropriate  measures  is  a 
daunting  task,  one  described  as  taking  years,  being 
intrinsically  difficult,  and  requiring  a  disciplined 
approach  to  research  and  development  of  needed  measures. 
Philip  G.  Joyce  observes  that: 

National  defense  is  a  particularly  difficult  area 

for  which  to  develop  measures  of  ultimate 


53  United  States.  General  Accounting  Office,  Results-Oriented 
Government:  GPRA  Has  Established  a  Solid  Foundatuon  for  Achieving 
Greater  Results,  270. 

54  United  States.  General  Accounting  Office,  Status  of  Achieving 
Outcomes  and  Addressing  Major  Management  Challenges,  (Washington  D.C.: 
U.S.  General  Accounting  Office,  2001),  36. 

55  Larry  C.  Juul,  DOD  Implementation  of  the  Government  Performance  and 
Results  Act  of  1 993, Anonymous  (Virginia:  Defense  Logistic  Agency,  1997) 
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performance,  perhaps  because  it  is  almost 
impossible  to  determine,  without  some  military 
conflict,  whether  the  objective  of  national 
defense  is  being  achieved.  Measurement  is  further 
complicated  by  the  concept,  as  some  would  argue 
that  the  purpose  of  defense  is  not  winning  wars 
but  preventing  them.  For  this  reason,  many 
measures  of  performance  for  national  defense 
often  focus  on  the  quality  and  readiness  of 
military  forces,  assuming  that  a  well  qualified, 
appropriately  trained  military  can  best  achieve 
whatever  goals  are  paramount  at  the  moment. 56 

However,  the  efforts  to  align  performance  measures 
into  budget  would  be  much  easier  because  DOD  has  already 
implemented  the  PPBS  which  provides  a  decision-making 
framework  of  budget  formulation  process  and  a  basis  of 
developing  performance  measurement  system. 

Since  the  earliest  implementation  of  GPRA,  The 
Department  of  Defense  has  tried  to  incorporate  the  budget 
development  in  the  context  of  0MB  guidance  and  in  the 
context  of  its  own  budget  development  and  resource 
allocation  decision  making  process  in  PPBS.  Explicitly,  the 
Under  Secretary  of  Defense  (Comptroller)  ,  who  is  also  the 
person  responsible  for  overseeing  the  implementation  of 
GPRA  within  DOD,  clearly  outlined  in  a  1995  memorandum  that 
general  guidelines  for  DOD  implementation  include  that  the 
plans  reflect  the  most  important  aspects  of  Defense;  have 
DOD  consensus;  are  integrated  with  the  DOD  Planning, 
Programming,  and  Budgeting  System  (PPBS) ,  and  are  supported 
by  functional  managers  who  will  be  responsible  for 
achieving  the  goals.  57 

56  Joyce,  1993  In  ibid. 

57  Office  of  the  Under  Secretary  of  Defense,  Comptroller,  1995. 
Department  of  Defense  Corporate  Level  Performance  Goals  and  Measures 
under  the  GPRA  -  ACTION  MEMORANDUM,  Washington,  DC  (April  24)  In  ibid. 
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Initially,  the  DOD  implemented  the  GPRA  in  two  ways  by 
sponsoring  performance  measurement  pilot  activities  and  by 
developing  and  securing  internal  consensus  on  DOD-wide 
corporate  goals.  Beginning  in  1994,  DOD  also  initiated  to 
develop  a  set  of  DOD-wide  corporate  goals  as  part  of  GPRA 
implementation.  Several  DOD  activities  became  the  pilot 
projects:  Defense  Logistics  Agency,  Army  Audit  Agency,  Army 
Corps  of  Engineers  (Civil  Works) ,  Defense  Commissary 
Agency,  Air  Combat  Command,  Navy  Carrier  Battle  Group 
(CINCLANTFLT) ,  and  Army  Research  Laboratory. 

In  1995  DOD  made  several  improvements  toward  the 
effort  implementing  the  GPRA  such  as  providing  draft  DOD 
mission  and  vision  statements,  and  seven  corporate  goals 
were  provided  to  the  Joint  Chiefs  of  Staff  (JCS)  and 

Under/Assistant  Secretaries  of  Defense  and  issuing  a  new 
unclassified  Annex  A  to  the  Defense  Planning  Guidance 
(DPG)  ,  one  of  major  documents  in  the  PPBS  cycle  and  a 

classified  set  of  assumptions  and  instructions  developed  by 
the  Office  of  the  Assistant  Secretary  of  Defense  (Strategy 
and  Requirements)  and  approved  by  the  Deputy  Secretary  of 
Defense  that  the  Components  follow  in  preparing  their 

Program  Objectives  Memorandum  (POM)  and  budget.  The  most 

important  idea  in  the  unclassified  DPG  annex  was  the 
development  of  draft  performance  measures  for  use  in  the 

DOD  DPRA  performance  plan. 

The  instructions  to  the  Military  Services  for 
preparation  of  the  Department's  FY  1997-2001  Program 

Objectives  Memorandum  (POM)  issued  in  February  of  1995 
included  the  following  requirement  "Components  should 
review  the  DOD  corporate  level  goals  and  develop 

performance  measures  that  relate  to  them.  During  the 
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program 

review. 

components 

will  provide  a  report  to  the 

Program 

Review 

Group 

that 

identifies  these  performance 

measures 

.  "  58 

In 

April 

1996, 

GPRA 

implementation  was  further 

escalated  by 

incorporating 

a  GPRA  annex  in  DPG  and 

reflecting  DOD  mission,  vision,  and  the  six  corporate  level 
goals  refined  in  the  instructions  for  the  FY  1998-2003  POM. 
These  will  be  adjunct  to  the  National  Military  Strategy  and 
Quadrennial  Defense  Review  which  are  intended  to  be  used  to 
satisfy  the  strategic  planning  requirements  of  the  GPRA. 
Specific  DOD  mission,  vision  and  corporate  level  goals  have 
been  developed  and  a  current  published  rendition  appears 
below :  59 


Table  2:  United  States  DOD  Mission,  Vision,  Goals 


DOD  Mission 

Statement 

The  mission  of  the  Department  of  Defense  is  to  support  and 
defend  the  Constitution  of  the  United  States,  to  provide 
for  the  common  defense  of  the  United  States,  its  citizens 
and  its  allies,  and  to  protect  and  advance  U.S.  interest 
around  the  world 

DOD  Vision 

Statement 

The  Department  of  Defense: 

•  Successfully  defends  the  United  States,  its  citizens, 
interests  and  allies 

•  Fields  the  best  trained,  best  equipped,  best  prepared 
joint  fighting  force  in  the  world 

•  Supports  alliance/security  relationships  that 

protect/advance  U.S.  security  interests 

•Advances  national  priorities  in  concert  with  Congress, 
other  agencies  and  the  private  sector 

•  Serves  as  a  model  of  effective,  efficient  and  innovative 
management  and  leadership 

DOD  Corporate 

Level  Goals 

1.  Provide  a  flexible,  ready,  and  sustainable  military 
force  capable  of  conducting  joint  operations  to  execute 
the  National  Military  Strategy,  including  the  capability 
to : 

•  Rapidly  project  power  to  deter  and,  if  necessary, 
fight  and  win  two  nearly  simultaneous  major  regional 
conflicts  (MRCs)  in  concert  with  regional  allies. 

•  Support  friends  and  allies  and  shape  the 

international  environment  in  ways  favorable  to  U.S. 
interests  through  peacetime  overseas  presence 

58  Office  of  the  Secretary  of  Defense,  Program  Analysis  and  Evaluation, 
1995.  FY  1997-2001  POM  Preparation  Instructions.  Washington,  DC.  ibid. 

59  Maroni,  Alice  C.,  1996.  "DOD  Implementation  of  the  Government 
Performance  and  Results  Act,"  Armed  Forces  Comptroller,  Vol.  4,  No.  4, 
(Fall)  In  ibid. 
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•  Conduct  a  wide  range  of  contingency  operations, 
including  peace  operations,  counter-drug  and  counter¬ 
terrorist  activities,  noncombatant  evacuations,  and 
disaster  assistance. 

•  Deter,  prevent  and  defend  against  the  effective  use 
of  nuclear,  biological,  and  chemical  weapons  and  their 
delivery  means  and  prevent  the  acquisitions  of  such 
weapons . 

2.  Recruit  and  retain  well-qualified  military  and  civilian 
personnel  and  provide  them  with  equal  opportunity  and  a 
high  quality  of  life. 

3.  Sustain  and  adapt  security  alliances,  enhance  coalition 
war  fighting,  and  forge  military  relationships  that 
protect  and  advance  U.S.  security  interests. 

4.  Maintain  U.S.  qualitative  superiority  in  key  war 
fighting  capabilities  (e.g.  information  warfare, 
logistics) . 

5.  Reduce  costs  and  eliminate  unnecessary  expenditures, 
while  maintaining  required  military  capability  across 
all  DOD  mission  areas  by  employing  all  modern  management 
tools  and  working  closely  and  effectively  with  other 
government  agencies.  Congress,  and  the  private  sector. 


By  launching  the  Defense  Reform  Initiative  report  in 
1997,  a  reform  plan  for  business  operations,  DOD 
established  a  management  oversight  structure  to  help 
sustain  the  direction  and  emphasis  of  these  efforts.  The 
requirement  to  develop  annual  performance  contracts  was 
started  in  fiscal  year  1999.  The  Defense  Reform  Initiative 
was  such  a  performance  contracts  which  were  internal 
management  agreements  between  the  principal  staff 
assistants— also  known  as  Under  and  Assistant  Secretaries— in 
the  Office  of  the  Secretary  of  Defense,  the  Deputy 
Secretary,  and  the  defense  agencies. 

By  this  performance  contract,  DOD  agencies  were 
required  to  improve  goals  and  objectives  in  terms  of  cost, 
productivity,  quality,  and  responsiveness  to  customers  and 
to  assess  the  progress  of  achieving  performance  contract 
goals  in  annual  performance  reports.  DOD  intended  the 
performance  contracts  to  improve  oversight  of  the  defense 
agencies  by  (1)  identifying  programming  and  budgeting 
issues,  (2)  providing  agency  directors  with  clear 


40 


objectives  and  performance  targets,  (3)  establishing  open 
and  direct  communications  with  customers  to,  among  other 
things,  demonstrate  how  their  actions  affect  agency 
performance,  and  (4)  changing  the  way  DOD  does  business. 60 

To  support  the  effort,  two  agencies  were  formed.  First 
was  the  Defense  Management  Council  to  oversee  Defense 
Reform  Initiative  efforts— chaired  by  the  Deputy  Secretary 
and  consisting  of  key  civilian  and  military  leaders.  Second 
was  the  Defense  Agency  Task  Force  to  oversee  the 
development  and  review  of  performance  contracts— chaired  by 
the  Deputy  Director  of  Resource  Analysis  in  the  Office  of 
the  Secretary  of  Defense,  Program  Analysis  and  Evaluation 
(OSD/PA&E)  and  consisting  of  senior-level  representatives 
from  the  Under  Secretaries  of  Defense,  the  service 
secretaries,  and  the  Joint  Staff. 

Eour  defense  agencies:  the  Defense  Logistics  Agency, 
the  Defense  Finance  and  Accounting  Service,  the  Defense 
Contract  Audit  Agency,  and  the  Defense  Health  Program  were 
selected  to  implement  performance  contracts  during  fiscal 
year  1999,  and  gradually  expanded  the  requirement  to  a 
total  of  10  agencies  by  fiscal  year  2002.  In  2001 
Quadrennial  Defense  Review  Report,  DOD  uses  the  risk 
management  framework  to  consider  trade-offs  among  defense 
objectives  and  resource  constraints.  The  framework  consists 
of  four  dimensions  of  risk: 

•  Force  management— the  ability  to  recruit, 

retain,  train,  and  equip  sufficient  numbers  of 
quality  personnel  and  sustain  the  readiness  of 


60  United  States.  Government  Accountability  Office,  Defense  Management: 
Tools  for  Measuring  and  Managing  Defense  Agency  Performance  Could  be 
Strengthened,  (Washington  D.C.:  U.S.  Government  Accountability  Office, 
2004),  53. 
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its  many 


the  force  while  accomplishing 
operational  tasks; 

•  Operational— the  ability  to  achieve  military 
objectives  in  a  near-term  conflict  or  other 
contingency; 

•  Future  challenges— the  ability  to  invest  in  new 
capabilities  and  develop  new  operational 
concepts  to  dissuade  or  defeat  mid-  to  long¬ 
term  military  challenges;  and 

•  Institutional— the  ability  to  develop  management 
practices  and  controls  that  use  resources 
efficiently  and  promote  the  effective  operation 
of  the  defense  establishment . 6i 

To  develop  performance  goals  and  report,  DOD  also  gave 
the  defense  agencies  the  option  to  use  a  balanced 
scorecard.  According  to  GAO,  DOD  requires  10  of  27  defense 
agencies  with  businesslike  missions  to  have  either 
performance  plans  or  balanced  scorecards,  five  defense 
agencies  use  performance  plans,  and  the  remainder  have 
exercised  the  option  to  develop  balanced  scorecards.  In 
addition,  three  intelligence  agencies  have  developed 
performance  plans  in  order  to  demonstrate  how  they  are 
coordinating  with  DOD  on  strategic  and  budgetary  planning 
in  the  post-September  2001  environment. 

Despite  the  above  effort,  GAO  still  feels 
unsatisfactory  with  DOD  efforts.  Based  on  its  2001  report62, 
GAO  claims  that  DOD' s  progress  in  achieving  the  selected 
outcomes  is  unclear,  because  most  of  the  selected  program 
outcomes  are  complex  and  interrelated  and  may  require  a 
number  of  years  to  accomplish,  and  also  because  DOD  did  not 


61  United  States.  Government  Accountability  Office,  Defense  Management: 
Tools  for  Measuring  and  Managing  Defense  Agency  Performance  Could  be 
Strengthened,  (Washington  D.C.:  U.S.  Government  Accountability  Office, 
2004),  53. 

62  United  States.  General  Accounting  Office,  Status  of  Achieving 
Outcomes  and  Addressing  Major  Management  Challenges ,  36. 
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provide  a  full  assessment  of  its  performance.  GAO  continues 
by  identifying  some  weaknesses  in  DOD  strategies  for 
achieving  unmet  performance  goals  in  the  f uture : 63 


Table  3:  Weaknesses  in  DOD  strategies  for  achieving  unmet 
performance  goals  in  the  future 


Planned  Outcome 

Weaknesses 

The  U.S.  maintain 
technological 
superiority  in  key  war¬ 
fighting  capabilities 

•  The  extent  to  which  DOD  has  made  progress  is 
unclear 

•  Some  of  the  performance  goals  underlying  measures- 
such  as  procurement  spending  and  defense  technology 
objectives-  do  not  provide  a  direct  link  toward 
meeting  the  goal  and  make  it  difficult  to  assess 
progress 

•  The  performance  report  does  not  reflect  concerns 
raised  about  the  adequacy  of  its  strategy  and  the 
timely  introduction  of  new  technologies  to 
operational  forces. 

U.S.  military  forces 
are  adequate  in  number, 
well  qualified,  and 
highly  motivated 

•  Performance  measures  do  not  adequately  indicate  its 
progress  toward  achieving  the  outcome  of  ensuring 
that  U.S.  military  forces  are  adequate  in  number, 
well  qualified,  and  highly  motivated. 

•  With  the  exception  of  the  enlisted  recruiting  area, 
DOD  does  not  identify  specific  planned  actions  that 
it  or  the  services  will  take  to  assist  them  in 
meeting  unmet  performance  targets  in  the  future. 

•  The  report  did  not  discuss  the  difficulties  of  the 
current  retention  environment,  but  the  report 
contains  little  clear  articulation  of  specific 
actions  or  strategies  being  taken  to  improve  future 
retention 

Combat  readiness  is 
maintained  at  desired 
levels 

•  The  level  of  progress  maintaining  combat  readiness 
at  desired  levels  is  unclear 

•  Some  performance  measure  targets  had  been  lowered, 
were  incomplete,  or  were  not  met  which  make 
difficult  to  assess  the  progress. 

Infrastructure  and 
operating  procedures 
are  more  efficient  and 
cost  effective 

•  The  extent  of  the  progress  to  be  more  efficient  and 
cost-effective  is  unclear 

•  The  targets  such  as  disposing  of  property,  reducing 
of  its  logistics  response  time,  and  streamlining 
the  acquisition  workforce  do  not  hold  up  to 
scrutiny  and  some  targets  had  been  lowered  or  were 
not  met. 

Reduced  availability 
and/or  use  of  illegal 
drugs 

•  DOD  FY2000  performance  report  did  not  include 
performance  goals  or  measures  related  to  its 
outcome,  therefore  no  basis  to  assess  the  progress. 

Fewer  erroneous 
payments  to  contractors 

•  DOD  had  no  performance  goals  directly  related  to 
achieving  the  outcome. 

63  United  States.  General  Accounting  Office,  Status 
Outcomes  and  Addressing  Major  Management  Challenges ,  36. 
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of  Achieving 


In  February  and  August  2003,  DOD  issued  guidance  to 
refine  its  approach  to  performance  management  and  renamed 
the  contracts  as  performance  plans.  First  implemented  in 
September  2003,  according  to  GAO,  the  performance  plans 
build  on  the  performance  goals  and  measures  established  in 
the  performance  contracts  and  are  intended  to  include  a 
more  comprehensive  set  of  performance  measures  that  align 
with  agency  strategic  plans  and  department-wide  goals  set 
forth  in  DOD' s  risk  management  framework. 

In  2004  report, 64  based  on  its  review  on  three  defense 
agencies  DLA,  DISA,  and  DODEA,  GAO  acknowledges  that  the 
performance  contracts  or  performance  plans  were  an 
important  step  toward  more  results-oriented  management 
practices  for  the  defense  agencies,  however,  the  contracts 
would  have  been  more  effective  tools  for  informing  decision 
making  and  managing  and  measuring  agency  performance  had 
the  agencies  included  additional  attributes  associated  with 
results-oriented  management.  GAO  further  observes  that 
contracts  showed  progress  made  and  encouraged  discussion 
among  top  agency  and  OSD  leaders  about  actions  needed  to 
achieve  targeted  results,  particularly  in  terms  of  quality, 
quantity,  cost,  and  timeliness  of  services,  however,  DOD 
guidance  did  not  require  performance  contracts  to  include 
certain  key  attributes  to  be  associated  with  results- 
oriented  management. 

The  Mercatus  Center,  George  Mason  University,  which 
has  developed  a  seven-step  process,  called  "Outcome-Based 
Scrutiny, "  that  provides  a  framework  for  comparing  the 


64  United  States.  Government  Accountability  Office,  Defense 
Management:  Tools  for  Measuring  and  Managing  Defense  Agency  Performance 
Could  be  Strengthened,  53. 
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results  and  costs  of  programs  with  similar  objectives  and 
assessing  the  likely  impact  of  reallocating  resources  to 
the  most  effective  programs  had  assessed  the  agencies 
annual  performance  report  scorecard.  The  purpose  of  the 
assessment  is  not  to  ascertain  how  well  the  agencies' 
reports  inform  the  public  about  the  results  they  produced 
so  that  policymakers  and  citizens  may  make  informed 
judgments  about  the  agencies'  results.  The  evaluation 
factors  are  grouped  under  three  general  categories;  does 
the  agency  report  its  accomplishments  in  a  transparent 
fashion;  does  the  report  focus  on  documenting  tangible 
public  benefits  the  agency  produced;  does  the  report  show 
evidence  of  forward-looking  leadership  that  uses 
performance  information  to  devise  strategies  for 
improvement  ? 

In  general,  based  on  its  assessment  on  FY  2003  and 
2004,  DOD  ranked  as  one  of  the  lowest  agencies  which  inform 
the  performance  to  the  public.  The  scoring  comparison  of 
DOD  to  other  federal  agencies  for  FY  2003  and  FY  2004  can 
be  seen  in  following  table: 
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Table  4 :  Scoring  Comparison  Between  DOD  and  Other  Federal 
Agencies  for  FY  2003  and  FY  2004 


Government  Departments/Agencies 

FY  2003  ScoringfiS 

FY  2004  Scoring66 

T 

PB 

L 

Rank 

T 

PB 

L 

Rank 

Department  of  Labor 

18 

16 

14 

1 

17 

17 

16 

1 

Department  of  State 

15 

14 

14 

4 

18 

17 

14 

2 

Department  of  Transportation 

16 

15 

17 

1 

17 

16 

13 

3 

Department  of  Veterans  Affairs 

17 

15 

15 

3 

17 

15 

14 

3 

Department  of  Commerce 

10 

11 

9 

16 

14 

11 

15 

5 

Small  Business  Administration 

10 

15 

11 

7 

13 

13 

13 

6 

Department  of  Justice 

10 

10 

11 

15 

14 

12 

13 

6 

Department  of  Energy 

13 

9 

10 

12 

14 

10 

15 

6 

United  States  Agency  for  International 
Development 

8 

9 

7 

21 

11 

14 

12 

9 

Environmental  Protection  Agency 

13 

11 

12 

7 

14 

12 

10 

10 

Nuclear  Regulatory  Commission 

13 

13 

10 

7 

13 

12 

11 

10 

National  Science  Foundation 

10 

9 

9 

17 

13 

12 

11 

10 

Department  of  the  Interior 

13 

15 

12 

6 

11 

12 

12 

13 

Department  of  Agriculture 

15 

13 

15 

4 

14 

10 

11 

13 

Department  of  Education 

11 

10 

11 

12 

13 

10 

11 

15 

Department  of  Housing  and  Urban 
Development 

15 

9 

11 

10 

10 

11 

11 

16 

Department  of  the  Treasury 

10 

8 

9 

19 

10 

12 

10 

16 

General  Services  Administration 

14 

11 

10 

10 

12 

9 

11 

16 

National  Aeronautics  and  Space 
Administration 

9 

7 

9 

20 

10 

8 

14 

16 

Social  Security  Administration 

13 

8 

11 

12 

12 

8 

12 

16 

Office  of  Personnel  Management 

9 

8 

11 

17 

9 

8 

13 

21 

Department  of  Homeland  Security 

NA 

NA 

NA 

NA 

7 

9 

11 

22 

Department  of  Defense 

11 

6 

6 

22 

7 

6 

9 

23 

Department  of  Health  and  Human  Services 

8 

7 

8 

22 

NA 

NA 

NA 

NA 

Specifically,  for  FY  2003,  the  Mercatus  Center 
observes  that  for  transparency,  although  the  report  could 
be  located  on  DOD  website  and  the  performance  information 
is  confined  to  30  pages  in  a  400  page  document.  Management 
Discussion  and  Analysis  section  lists  40  systemic  and 
material  weaknesses  in  the  performance  data  and  an  appendix 
contains  baseline  and  trend  data  back  to  fiscal  1999  or 
2000  for  each  goal. 

For  public  benefits,  first,  three  of  four  strategic 
goals,  and  about  half  of  the  13  performance  goals  are 


65  Mercatus  Center.  George  Mason  University,  5th  Annual  Performance 
Report  Scorecard :  Which  Federal  Agencies  Best  Inform  the  Public  (FY 
2003 j , Anonymous  George  Mason  University.  Mercatus  Center,  2004) 

66  Mercatus  Center.  George  Mason  University,  6th  Annual  Performance 
Report  Scorecard :  Which  Federal  Agencies  Best  Inform  the 
Public , Anonymous  George  Mason  University.  Mercatus  Center,  2005) 
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outcomes;  second,  performance  measures  do  not  begin  to 
capture  the  breadth  and  depth  of  Defense's  operations  and 
do  not  measure  key  results  related  to  strategic  goals; 
third,  there  is  no  discussion  of  the  recent  operations  in 
Iraq  and  Afghanistan;  and  fourth,  lack  of  information  about 
the  department's  actual  performance;  and  finally, 
voluminous  financial  information  is  not  linked  to  strategic 
goals  or  measures. 

For  leadership,  first,  performance  shortfalls 
disclosed  with  minimal  explanation;  second,  senior 
management  offers  no  direct  response  to  nine  major 
management  challenges  identified  by  the  inspector  general, 
except  some  proposed  legislative  changes,  third,  six 
Defense  issues  remain  on  General  Accounting  Office's  "High 
Risk  List.";  and  fourth,  the  efforts  are  underway  to 
transform  the  department,  but  offers  few  specifics. 

For  2004,  for  transparency,  the  Mercatus  Center 
observes  that  first,  the  report  is  available  via  an 
unintuitive  link  under  "DOD  Websites."  downloadable  as  a 
single  PDF  document  or  separate  file;  second,  confusing  mix 
of  risk-management  strategic  objectives,  "policy  goals," 
and  "performance  metrics.";  third,  report  omits  legally 
required  assessment  by  agency  head  of  data  shortcomings  and 
efforts  to  address  them,  although  portions  of  the  report 
make  clear  that  the  department  faces  severe  data  problems; 
fourth,  better  explanation  is  needed  for  baseline  and 
trend  data  back  to  fiscal  2001. 

For  public  benefits,  first,  four  strategic  objectives 
are  outcome-oriented,  and  goals  and  measures  are  a  mix  of 
outcomes  and  activities;  second,  department  states  that  the 
Quadrennial  Defense  Review  Report  serves  as  its  strategic 


47 


plan;  third,  existing  measures  do  not  come  close  to 
capturing  the  full  range  of  the  department's  operations; 
fourth,  the  department  is  in  the  process  of  developing  many 
new  performance  measures;  and  finally,  no  information  links 
goals  and  results  to  costs. 

For  leadership,  first,  report  says  little  about  the 
department's  most  obvious  accomplishments;  second,  when 
performance  shortfalls  are  acknowledged,  accompanying 
explanations  are  often  perfunctory,  third.  Inspector 
general  says  the  department  has  made  progress  on  major 
management  challenges  but  provides  few  specifics;  fourth, 
some  initiatives  to  improve  on  specific  objectives  are 
described;  and  finally.  Deputy  secretary's  letter  says  11 
management  control  weaknesses  were  corrected,  but  46 
remain . 

H.  COMPARATIVE  EXPERIENCE  WITH  OTHER  COUNTRIES 

1 .  Britain 

The  performance  initiative  was  started  by  launching 
the  Financial  Management  Initiatives  in  1982  which  aims  to 
quantify  wherever  feasible  the  assessment  of  whether 
programs  are  providing  "value  for  money."  It  was  introduced 
when  Britain  experienced  a  severe  pressure  on 
administrative  expenditure  and  contraction  in  public 
employment.  In  announcing  FMI,  the  government  sought  "in 
each  department  an  organization  and  a  system  in  which 
managers  at  all  levels  have  a  clear  view  of  their 
objectives  and  the  means  to  assess,  and  wherever  possible. 
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measure  outputs  or  performance  in  relation  to  those 
ob j  ectives  .  67 

McCaffery  and  Jones  observe  that: 

The  Financial  Management  Initiative  called  for  a 
radical  change  in  the  internal  structure  and 
operations  of  government  agencies.  Objectives 
were  to  be  assigned  to  responsibility  centers. 

Costs  were  to  be  systematically  identified.  They 
were  to  be  measured  on  and  accrual  bass  (i.e. 

matching  resources  consumed  to  services 
delivered)  and  include  not  only  the  direct  costs 
of  service  delivery  but  overheads  as  well.  This 
identification  enabled  those  responsible  for 
meeting  particular  objectives  to  be  held 
accountable  for  the  cost  or  the  resources  they 

were  consuming. 68 

FMI  does  not  require  a  uniform  approach  for  all 
departments.  Although  there  is  a  common  element,  an 

insistence  on  delegated  budgeting  in  which  responsibility 
for  resources  is  pushed  down  the  line  to  budget  holders 
(those  who  actually  spend  resources  and  carry  out 

operations)  who  are  to  be  given  sufficient  flexibility  and 
incentive  to  produce  value  for  money69,  each  department  is 
freely  developing  the  managerial  style  and  system  suited  to 
its  circumstances  as  long  as  it  is  in  accordance  with 
certain  broad  guidelines. 

FMI  conceives  of  budgeting  as  a  "contract  for 
performance"  in  which  departments  commit  themselves  to 
concrete  targets  in  exchange  for  agreed  resources.  A  1986 


61  Cmnd.  8616,  op.cit.  in  Allen  Schick,  "Budgeting  for  Results:  Recent 
Developments  in  Five  Industrialized  Countries,"  Public  administration 
review  50,  no.  1  (1990) :  26-34. 

68  McCaffery  and  Jones,  Budgeting  and  financial  management  in  the 
federal  government,  472. 

69  hM  Treasury,  Multi-departmental  Review  of  Budgeting:  Final  Central 
Report,  1986,  p.  8  in  Schick,  Budgeting  for  Results:  Recent 
Developments  in  Five  Industrialized  Countries,  26-34. 
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Treasury  report  defined  budgeting  as  a  means  of  delivering 
value  for  money  against  a  background  of  aims,  objectives 
and  targets  and  it  will  only  fully  realize  its  full 
potential  if  it  has  the  support  and  involvement  of 
management,  if  there  are  strong  connections  between 
budgets,  outputs  and  results,  and  if  it  operates  within  a 
supportive  central  and  managerial  environment . 

The  latest  major  development  of  central  government 
performance  measurement  is  Public  Service  Agreements  (PSA) , 
which  was  launched  in  1998  and  represents  a  fundamental 
change  in  the  accountability  of  government  to  Parliament 
and  the  public.  Each  department  (ministry)  is  required  to 
provide  its  own  PSA.  Each  PSA  sets  out  the  key  performance 
targets  which  will  contribute  to  achieving  aims  and 
objectives  and  includes  an  introduction  which  explains  how 
these  contribute  to  the  Government's  overall  strategy  to 
meet  its  key  Comprehensive  Spending  Review  (CSR)  objectives 
and  to  improve  productivity. 

PSA  includes  in  single  document  important  information 
on  aims  and  objectives,  resources,  performance  and 
efficiency  targets  and  related  policy  initiatives.  It  has 
five  key  elements: 

•  An  introduction,  setting  out  the  Minister  or 

Ministers  accountable  for  delivering  the 

commitments,  together  with  the  coverage  of  the 
PSA,  as  some  cover  other  departments  and 
agencies  for  which  the  relevant  Secretary  of 
State  is  accountable; 

•  the  aims  and  objectives  of  the  department  or 
cross-cutting  area; 

•  the  resources  which  have  been  allocated  to  it 
in  the  CSR; 


70  hM  Treasury,  Multi-departmental  Review  of  Budgeting:  Executive 
Summary,  1986,  in  ibid. 
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•  key  performance  targets  for  the  delivery  of  its 
services,  together  with,  in  some  cases,  a  list 
of  key  policy  initiatives  to  be  delivered; 

•  a  statement  about  how  the  department  will 
increase  the  productivity  of  its  operations . 

By  this  approach,  the  government  is  required  to  measure 
and  improve  the  public  sector's  efficiency  and  thus  raise 
its  productivity,  the  quality  and  quantity  of  output 
delivered  for  the  financial  investment  made.  Each  PSA  will 
be  monitored  by  the  Government  to  meet  every  target.  Should 
a  target  not  be  met  there  is  no  question  of  money  being 
deducted  from  the  budget  for  that  department?  Nor  will 
additional  funding  over  and  above  that  already  allocated  be 
made  available  simply  because  a  department  is  failing  to 
meet  its  targets,  but  support  and  advice  will  be  given  by 

the  committee. 72 

To  compliment  the  PSA,  in  1998,  the  British  Government 
also  launched  the  Regulatory  Impact  Assessments  (RIA) 
program  to  assess  the  likely  outcomes  of  regulating  in 
order  to  try  to  achieve  the  right  balance  between  under¬ 
regulating,  which  may  fail  to  protect  the  public,  and  over¬ 
regulating,  which  may  create  excessive  bureaucracy.  RIA 
identifies  the  costs  and  benefits  of  a  policy  proposal  and 
the  risks  of  not  acting  and  is  intended  to  inform  the 
policy  decision  making  process  and  communicate  clearly  the 
objectives,  options,  costs,  benefits  and  risks  of  proposals 
to  the  public  to  increase  the  transparency  of  the  process. 


71  "Public  Services  for  the  Future:  Modernisation,  Reform, 
Accountability,  Comprehensive  Spending  Review:  Public  Service 
Agreements  1999-2002,"  in  The  Stationery  Office  [database  online]. 
Available  from  http :/ /www . archive . of ficial- 
documents .co.uk/ document/ cm4 l/4181/psa-4181. htm. 

72  ibid. 
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There  are  three  stages  on  RIA.  First,  initial  RIA 

carries  out  at  the  early  policy  development  stage  and  helps 

identify  areas  where  departments  need  more  information. 

Second,  partial  RIA  which  builds  on  the  Initial  RIA  should 

accompany  the  formal  public  consultation  for  the  proposal. 

Finally,  final  RIA,  which  also  on  the  analysis  in  the 

Partial  RIA,  updates  it  in  the  light  of  consultation  and 

further  analysis  and  information.  The  Final  RIA  should 

accompany  legislation  when  presented  to  Parliament . ^3 

From  the  three  stages  above,  it  can  be  noted  that  the 

process  of  RIA,  not  only  provides  an  evaluation  on 

government  policies  but  it  is  also  accompanied  by  public 

involvement  to  obtain  a  soundness  of  analysis  to  the  public 

interest.  RIA  has  helped  contribute  to  better  policy  making 

and  reducing  the  costs  to  business  and  policy  makers  could 

pay  more  attention  to  some  aspects  of  preparing  RIAs  which 

includes  evaluating  a  range  of  options  (including  not 

regulating)  and  encouraging  self-regulation  where  feasible. 

From  an  examination  of  RIAs  in  13  departments  and  agencies, 

RIAs  have  been  used  in  accordance  with  good  practice  and 

departments  and  agencies  have  applied  the  RIA  process  to 

good  effect  so  that  they  achieve  the  five  principles  of 

good  regulation:  transparency,  proportionality,  targeting, 

consistency  and  accountability.  For  example: 

...consultation  on  the  basis  of  the  RIA  on  the 
proposed  regime  for  the  National  Minimum  Wage 
resulted  in  the  adoption  of  a  different  method  of 
implementation  that  avoided  £150  million  in 
unnecessary  administrative  costs  to  employers.  And 
a  RIA  on  new  pesticides  regulations  showed  that 


'3  "Good  Practice:  Regulatory  Impact  Assessments  -  Introduction 
to  RIAs"  [cited  2005]  .  Available  from 
http : / / WWW . nao .org.uk/ria/ria  introduction . htm 
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employers  would  incur  disproportionate  costs  from 
a  new  mandatory  training  requirement  which  were 
not  justified  by  the  benefits.  So  a  non-regulatory 
option  was  adopted  instead.  ^4 

2 .  Canada 

Requirement  to  measure  and  report  on  performance  in 
Canada  has  been  started  since  the  early  1970s.  The  effort 
has  been  strengthened  by  launching  the  Increased 
Ministerial  Authority  and  Accountability  (IMAA)  in  1986, 
which  demands  increasing  accountability  for  performance  and 
allows  departments  to  expand  authority  to  reallocate 
resources  within  approved  funding  levels  as  well  as  to 
carry  a  portion  of  their  capital  funds  into  the  next  fiscal 
year  budget.  The  MOU,  which  was  being  individually 
prepared  and  negotiated  in  each  department,  covers  a  three 
year  period  and  consists  of  targets  and  an  assessment  of 
performance  geared  to  the  targets. 

After  a  public  sector  reform  to  improve  the 
effectiveness  and  cost  efficiency  of  all  government 
programs  in  1994,  in  1996  the  "Getting  Government  Right" 
initiative  was  introduced  to  modernize  federal  programs  and 
has  been  confirmed  as  a  significant  change  to  a  performance 
based  management  theory.  Department  and  agencies  were 
encouraged  to  identify  result  commitments  and  to  measure 
and  use  performance  inf ormation . ^5  in  April  1999,  the 
Performance  Management  Programme  was  introduced  to  provide 
a  framework  for  equitable  performance  management  for 


'4  "National  Audit  Office  Press  Notice:  Better  Regulation  -  Making 
Good  Use  of  Regulatory  Impact  Assessments,"  in  NAO  [database  online], 
[cited  2005].  Available  from  http://www.nao.org.uk/pn/01- 

02/0102329. htm 

75  The  Comptroller  and  Auditor  General,  Measuring  the  Performance  of 
Government  Departments , Anonymous  ,  2001) 
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federal  executives  and  to  reward  the  achievement  of 
results.  To  assure  the  quality  information,  government 
departments  adopt  quality  management  as  part  of  overall 
strategy  and  conduct  internal  audits  and  evaluations  of  key 
policies  and  programs. ’^6 

Related  to  the  high  level  objectives  and  type  of 
measurement  the  measures  are  fairly  comprehensive  both  in 
terms  of  inputs,  outputs,  outcomes,  etc.  and  in  coverage  of 
the  public  service.  However,  concerns  continue  to  be  raised 
about  the  integration  of  results  with  management  values; 
more  comprehensive  and  accessible  reporting;  and 
development  of  targets  etc.  linking  federal,  provincial  and 

local  agencies.  77 

The  agency  which  is  responsible  for  managing  the 
overall  performance  management  in  Canada  is  the  Treasury 
Board  Secretariat  (TBS) .  Its  representatives  are  placed  in 
each  department  or  agency.  For  reporting  to  Parliament,  TBS 
shares  its  authority  with  federal  bodies,  within  which 
Federal  bodies  provide  Performance  and  Annual  Reports  to 
Parliament  and  TBS  collates  and  reports  specifically 
through  the  Annual  Report. 


3.  Australia 

Although  until  the  mid-1980s,  financial 
highly  centralized,  and  spending  departments 
opportunity  or  incentive  to  manage  their 
interest  in  performance  management  dates  back 


control  was 
had  little 
finances ,  78 
to  a  report 


76  The  Comptroller  and  Auditor  General,  Measuring  the  Performance  of 
Government  Departments , Anonymous  ,  2001. 

77  ibid. 

78  Schick,  Budgeting  for  Results:  Recent  Developments  in  Five 
Industrialized  Countries,  26-34. 
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on  Australian  Government  administration  in  1976  (this  was 
called  the  Coombs'  Report)  .’^9  The  development  of  performance 
measurement  system  in  Australia  was  initiated  in  1980' s 
when  the  government  launched  two  major  reforms:  the 

Financial  Management  Improvement  Program  (FMIP)  and  Program 
Management  and  Budgeting  (PMB) ,  which  seeks  to  shift  focus 
of  budgeting  from  inputs  used  to  the  results  achieved.  A 
1983  government  study  found  that  94  percent  of  its  senior 
managers  perceived  financial  management  as  merely  spending 
the  total  allocated  or  controlling  expenditures  against 
appropriations . 80  Although  FMIP  was  actively  being  promoted, 
it  failed  to  balance  the  need  for  central  direction  with 
FMIP  commitment  to  allow  each  department  to  set  up  its  own 
management  system. 

Program  Management  and  Budgeting  (PMB)  is  the  key  for 
identifying  each  organization's  objectives  and  placing  them 
into  a  framework  within  which  targets  can  be  set  for 
managers  at  all  levels. 8i  Schick  continues  by  observing  that 
the  program  structure  arrays  objectives  in  hierarchical 
order,  with  those  at  the  lower  levels  more  subject  to  being 
expressed  in  measurable  terms  than  those  at  the  higher 
levels.  The  development  of  Australia's  PMB  had  become  more 
complicated  since  many  difficulties  in  defining  aims  and 
using  performance  indicators  to  measure  results. 

79  Beryl  A.  Radin,  "A  comparative  approach  to  performance  management: 
Contrasting  the  experience  of  Australia,  New  Zealand,  and  the  United 
States,"  International  Journal  of  Public  Administration  26,  no.  12 
(2003) :  1355. 

80  Australian  Public  Service  Board  and  Department  of  Finance,  Financial 

Management  Improvement  Program;  Diagnostic  study,  February  1984;  in 

Schick,  Budgeting  for  Results:  Recent  Developments  in  Five 

Industrialized  Countries,  26-34. 

81  Australian  Public  Service  Board  and  Department  of  Finance,  Financial 

Management  Improvement  Program;  Diagnostic  study,  February  1984;  in 

ibid . 
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In  1988,  the  federal  government  initiated  a  program 
for  evaluative  reviews  of  all  policy  areas  and  ensured  that 
the  results  were  considered  in  budgetary  decision 
processes.  In  February  1993,  The  Management  Advisory  Board 
and  Management  Improvement  Advisory  Committee  issued  a 
paper  called  "Performance  Information  and  the  Management 
Cycle.  It  asserted  that  "performance  information  is  being 
used  in  both  program  management  and  central  decision¬ 
making,  including  in  the  budget  and  context;  and  the 
quality  of  performance  reporting  is  improving . "82 

To  coordinate  the  delivery  of  public  services,  the 
federal  government  created  'Centrelink'  in  1997,  an  agency 
functioned  as  the  agency  providing  'one-stop-shopping 
access  to  a  range  of  government  services.  More  recently  the 
introduction  of  accrual  based  accounting  and  budgeting,  so 
called  Output  Based  Budgeting,  has  been  aimed  at  providing 
high  level  performance  information  about  outputs  and 
outcomes,  linked  to  more  accurate  resource  accounting,  so 
that  broad  'cost  benefit'  judgments  could  be  made. 83 

Within  the  new  approach,  every  agency  is  responsible 
for  performance  management.  Changes  to  outputs  and 
performance  information  are  agreed  with  the  relevant 
Minister  and  do  not  require  clearance  from  the  Department 
of  Finance  and  Administration.  Each  department  or  agency 
issues  information  on  resources  consumed;  outputs  (price, 
quantity,  quality)  to  be  delivered  and  the  outcomes  to  be 
contributed;  actual  outcomes  and  the  contribution  outputs 

82  Management  Advisory  Board  Australia,  1993,  Performance  Information 
and  the  Management  Cycle,  AGPS,  Canberra.  In  Radin,  A  comparative 
approach  to  performance  management:  Contrasting  the  experience  of 
Australia,  New  Zealand,  and  the  United  States,  1355. 

83  The  Comptroller  and  Auditor  General,  Measuring  the  Performance  of 
Government  Departments , 
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made  to  them;  and  finally  the  distinction  between  outcomes 
produced  by  agencies  controlled  by,  and  those  administered 

on  behalf  of  government . 84 

Performance  is  reported  publicly  in  individual  agency 
Annual  Reports  to  Parliament.  Department  of  Finance  and 
Administration  has  role  in  advising  the  development  of 
performance  measures  and  indicators.  For  the  quality 
assurance  of  the  information,  agency  audit  committees  are 
responsible  for  internal  audit  of  performance  information 
and  no  general  requirement  for  audit  and  validation. 
However,  the  Australian  National  Audit  Office  (ANAO) 
performs  value  for  money  audits  including  resource  use, 
delivery  of  outputs  and  outcomes,  monitoring  systems  and 
compliance.  Some  systemic  barriers  to  performance 
improvement  in  Australia  are: 

a.  No  real  link  between  appropriations  and 
objectives.  The  federal  government  had  a  system 
of  so  called  program  budgeting  but  in  reality 
had  program  reporting  only.  The  budget  process 
and  the  legislative  appropriations  process  was 
largely  determined  by  history. 

b.  No  clear  relationship  between  the  budgeted 
performance  promises  and  the  actual  performance 
report  in  annual  reports. 

c.  Responsibility  for  program  definition  and 
reporting  was  devolved  almost  entirely  to 
departments  and  agencies,  with  little 
legislative  or  executive  oversight.  This  was 
reinforced  by  a  cabinet  system  in  which  the 
specific  cabinet  minister  charged  with 
responsibility  for  a  department  was  not  likely 
to  take  "orders"  from  a  peer. 85 


04  The  Comptroller  and  Auditor  General,  Measuring  the  Performance  of 
Government  Departments , 

85  Speech  by  Stephen  Bartos,  Australian  Department  of  Finance,  April 
26,  2000  In  Radin,  A  comparative  approach  to  performance  management: 

Contrasting  the  experience  of  Australia,  New  Zealand,  and  the  United 
States,  1355. 
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I .  CHAPTER  SUMMARY 

The  GPRA  has  been  a  prominence  best  practice  in 
performance  based  budgeting  implementation.  It  is  caused  by 
a  series  of  effort  of  performance  oriented  budgeting 
experiences  from  the  first  Hoover  Commission,  PPBS  MBO, 
ZBB,  and  finally  the  GPRA.  Albeit  a  lot  of  criticisms  in 
the  earlier  of  its  implementation,  GPRA  has  proved  itself 
to  be  sound  government  performance  evaluation  tool  facing 
enormous  complexities  in  the  public  sectors.  The  important 
aspects  that  make  GPRA  successful  are,  first,  it  is  enacted 
by  the  law  so  that  it  has  supported  by  both  President  and 
Congress;  second,  it  is  included  in  the  President 
Management  Agenda  in  which  President  George  W.  Bush  put 
this  effort  as  one  of  the  important  program  during  his 
presidency;  finally,  0MB  has  opened  the  involvement  of  the 
stakeholders  such  as  GAO,  Congress,  agencies,  academics,  to 
provide  any  feedback  towards  its  implementation  and  to 
enrich  the  development  of  the  elements  of  GPRA. 

In  the  DOD,  facing  the  complexities  such  as  defense  is 
particularly  difficult  area  for  which  to  develop  measures 
of  ultimate  performance  and  it  involves  over  $1  trillion  in 
assets,  budget  authority  of  about  $310  billion  annually, 
and  about  3  million  military  and  civilian  employees  and 
these  operations  represents  one  of  the  largest  management 
challenges  within  the  federal  government,  the  performance 
evaluation  tools  generated  from  the  PPBES  system  has  also 
provided  a  decision-making  framework  of  budget  formulation 
process  and  a  basis  of  developing  performance  measurement 
system.  Several  pilot  projects  such  as  Defense  Logistics 
Agency  and  the  Defense  Finance  and  Accounting  Service  can 
be  set  as  best  practice  in  defense  business. 
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Flexibility  is  one  of  the  key  implementation  in  DOD, 
in  which  after  having  the  framework  consists  of  four 
dimensions  of  risk:  force  management,  operational,  future 
challenges,  and  institutional,  DOD  gave  the  defense 
agencies  the  option  to  use  a  balanced  scorecard.  Quick  feed 
back  has  also  been  provided  in  implementing  the  GPRA.  For 
example,  facing  some  weaknesses  in  DOD  strategies  for 
achieving  unmet  performance  goals  in  the  future,  DOD 
quickly  replied  by  issuing  a  guidance  to  refine  its 
approach  to  performance  management  and  renamed  the 
contracts  as  performance  plans  that  intended  to  include  a 
more  comprehensive  set  of  performance  measures  that  align 
with  agency  strategic  plans  and  department-wide  goals  set 
forth  in  DOD' s  risk  management  framework. 

The  results  of  Outcome-Based  Scrutiny  tool  from  the 
Mercatus  Center,  George  Mason  University  probably  can  be 
used  as  the  third  party  judgment  towards  GPRA 
implementation,  because,  academically,  it  inform  the  public 
about  the  accountability  perspective,  transparency,  public 
benefits,  and  leadership,  of  the  GPRA  implementation 
results  produced  by  the  agencies  so  that  policymakers  and 
citizens  may  make  informed  judgments  about  the  agencies' 
results.  This  tool  provides  more  sound  on  how  government 
manages  every  Federal  tax  dollar  and  what  level  and  quality 
of  government  service  are  returned  to  the  taxpayer. 
Specifically,  DOD  still  ranks  as  one  of  the  lowest  agencies 
which  inform  the  performance  to  the  public. 
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Ill .  PERFORMANCE  BASED  BUDGETING 


A.  INTRODUCTION 

This  chapter  lays  out  the  literature  review  of 
Performance  Based  Budgeting  (PBB) ,  a  prominent  reform 
around  the  world.  Section  B  reviews  overview  of  PBB  about 
its  definitions  and  principle.  Section  C,  D,  and  E  provides 
the  literature  review  of  Performance  Based  Budgeting 
including  strategic  planning,  performance  measures  and 
linking  performance  to  budget.  A  summary  is  provided  at  the 
end  of  the  chapter  in  Section  F 

B .  OVERVIEW 

Budgeting  is  essentially  the  allocation  of  resources; 
it  inherently  involves  setting  priorities.  In  its  broadest 
sense,  the  budget  debate  is  the  place  where  competing 
claims  and  claimants  come  together  to  decide  how  much  of 
the  government's  scarce  resources  will  be  allocated  across 
many  compelling  national  purposes. 86 

In  traditional  line  item  budgeting  system,  government 
expenditure  is  set-up  based  on  line  items  or  major  object 
expenditure.  In  this  system,  budget  control  is  done  by 
stating  the  limits  of  spending  on  each  item  in  the  budget 
allocation.  For  the  addition  or  deletion,  incrementalism 
theory  has  played  role.  "Incrementalism"  is  widely  held 
budgetary  concept,  which  attributed  to  the  works  of  Aaron 
Wildavsky  and  further  developed  by  Richard  Fenno.  It  was  a 
bottom-up  method  of  developing  the  budget  through  a  series 


United  States.  General  Accounting  Office,  Performance  Budgeting; 
Current  Development  and  Future  Prospects,  (Washington  D.C.:  U.S. 
General  Accounting  Office,  2003),  24. 
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of  political  adjustments  that  tended  to  increase  the  budget 
base  in  small  controlled  incrementsS? 

In  practice,  line  item  budgeting  with  its 
incrementalism  concept  does  not  provide  information  about 
the  reasons  why  government  made  the  expenditures  or  even 
does  not  provide  the  aspects  of  efficiency  and 
effectiveness  of  the  programs.  Decisions  about  future 
spending  are  taken  without  sufficiently  examining  the 
outputs  or  outcomes  of  the  government  programs.  It  can  be 
said  that  without  having  an  adequate  examination  of 
efficiency  and  effectiveness  aspects,  government 
accountability  is  still  arguable.  Here,  performance  based 
budgeting  or  performance  budgeting  can  be  used  as  a  tandem 
to  complement  the  line  item  budget  system. 

However,  it  remains  questionable  whether  after 
performance  budgeting  is  implemented,  the  other  budgeting 
process  will  not  be  used  or  whether  the  performance 
information  is  the  only  consideration  in  formulating  the 
budget  and  allocating  the  resources.  While  GAO  states  that 
performance  budgeting  cannot  replace  the  budget  process  as 
it  currently  exists,  but  it  can  help  shift  the  focus  of 
budgetary  process  as  oversight  activities  by  changing  the 
agenda  of  questions  asked  in  these  processes ; 88  Phillip  G. 
Joyce, 89  on  the  other  hand,  states  that  agencies  may  replace 
traditional  line-item  budgets  with  a  system  that  grants 
program  managers  greater  flexibility  in  managing  their 
resources  but  holds  them  accountable  for  achieving  program 


Pitsvada  and  LoStracco,  Performance  budgeting--the  next  budgetary 
answer.  But  what  is  the  question?,  53. 

88  United  States.  General  Accounting  Office,  Performance  Budgeting: 
Current  Development  and  Future  Prospects,  24. 

89  Joyce,  Performance  Based  Budgeting,  597-619. 
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results.  He  continues  by  observing  that  even  if  government¬ 
wide  decisions  on  resource  allocation  are  not  affected, 
agencies  may  find  measures  that  are  valuable  for  improving 
their  management  of  a  given  level  of  resources,  regardless 
of  whether  the  use  of  those  measures  results  in  a 
significant  shift  of  resources  from  on  program  or  agency  to 
another.  Also,  the  government  or  particular  agencies  may 
use  performance  measures  as  a  part  of  reports  on  their 
accomplishments . 

In  general.  Performance  Based  Budgeting  is  a  set  of 
process  of  translating  strategic  planning  framework  into 
performance  measures  in  terms  of  input,  output,  and  outcome 
to  be  tied  into  resource  allocation  in  budgeting  process. 
The  concept  of  performance  budgeting  assumes  that  a 
systematic  presentation  of  performance  information 
alongside  budget  amounts  will  improve  budget  decision¬ 
making  by  focusing  funding  choices  on  program  results. 90 
Melkers  and  Willoughby  describes  performance  based 
budgeting  as  a  more  diverse  set  of  requirements,  blending 
various  aspects  of  current  public  management  trends, 
including  outcome  measurement,  performance  measurement 
systems,  strategic  planning  and  benchmarking . 9i  From 
definitions  above,  at  least,  three  principles  that  must  be 
established:  strategic  planning,  performance  measurement 
and  linking  performance  to  budget. 


90  United  States.  General  Accounting  Office.  Accounting  and 
Information  Management  Division,  Performance  budgeting  [microform] 
fiscal  year  2000  progress  in  linking  plans  with  budgets, Anonymous 
(Washington,  D.C.  (P.O.  Box  37050,  Washington,  D.C.  20013)  :  The  Office, 
1999),  11,  2. 

91  J.  E.  Melkers  and  K.  G.  Willoughby,  "Budgeters  Views  of  State 

Performance-Budgeting  Systems:  Distinctions  across  Branches,"  Public 
administration  review  61,  no.  1  (2001)  :  54-64. 
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c. 


STRATEGIC  PLANNING 


Strategic  planning  is  the  process  of  deciding  on 


ob j  ectives 

of 

the  organization. 

on 

changes 

in  these 

ob j  ectives , 

on 

the  resources 

used 

to 

attain  these 

ob j  ectives , 

and 

on  the  policies 

that 

are 

to 

govern  the 

acquisition,  use,  and  the  disposition  of  these  resources. 92 
It  blends  futuristic  thinking,  objective  analysis,  and 
subjective  evaluation  of  goals  and  priorities  to  chart 
future  courses  of  action  that  will  ensure  the  long-run 
vitality  and  effectiveness  of  the  organization . 93 

Strategic  planning  is  maybe  such  a  new  concept  in 
public  sector  management  which  has  been  more  familiar  with 
the  long  term  planning  concept.  The  successful  of  this  idea 
in  private  sector  has  brought  public  manager  attention  on 
how  to  apply  it  in  the  complex  world  of  public  sector. 
According  to  Robert  B.  Denhart,  94  the  reasons  for  public 
organizations  to  undertake  strategic  planning  efforts  are 
many  -  to  give  clarity  and  direction  to  the  organization, 
to  choose  from  among  competing  goals  and  activities,  to 
cope  with  expected  shifts  in  the  environment,  and  to  bring 
together  the  thoughts  and  ideas  of  all  participants  in  the 
work  of  the  organization. 


92  Schick,  Budgeting  for  Results:  Recent  Developments  in  Five 
Industrialized  Countries,  26-34. 

93  Poister,  Theodore  H.,  and  Gregory  D.  Streib,  "Strategic  Management 
in  the  Public  Sector,"  in  Performance  Based  Budgeting,  ed.  Miller, 
Gerald  J.,  W.  Bartley  Hildreth,  and  Jack  Rabin.  Anonymous  (Colorado: 
Westview  Press,  2001),  283-305. 

94  Robert  B.  Denhardt,  "Strategic  Planning  in  State  and  Local 
Government,"  in  Performance  Based  Budgeting,  ed.  Miller,  Gerald  J.,  W. 
Bartley  Hildreth,  Jack  Rabin.  Anonymous  (Colorado:  Westview  Press, 
2001),  233-244. 
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A  strategic  planning  is  different  with  a  long  range 


planning.  Robert  B.  Denharts^  points  three  ways  why 
strategic  planning  is  different  with  a  traditional  long 
range  planning.  First,  while  traditional  planning 
activities  are  concerned  primarily  with  establishing  goals 
or  objectives  over  a  period  of  time,  but  less  concerned 
with  specific  steps  that  should  be  undertaken  to  achieve 
those  goals,  strategic  planning  consists  of  a  series  of 
action  steps  will  develop  as  part  of  the  planning  process 
and  activities  guidance  for  organization  in  the  immediate 
future  in  which  it  takes  the  future  into  account,  but 
improves  decisions  in  the  present. 

Second  is  its  attention  to  environmental  complexity  in 
which  the  organization  is  not  assumed  to  exist  in  a  vacuum, 
but  rather  both  the  organization's  objectives  and  steps  to 
achieve  those  objectives  are  seen  in  the  context  of  the 
resources  an  constraints  presented  by  the  organization's 
environment . 96  The  last  distinction  is  that  strategic 
planning,  especially  in  the  public  sector,  is  a  process 
that  must  involve  many  individuals  at  many  levels  of  the 
organization . 

In  designing  the  components  of  strategic  planning, 
there  are  two  approaches  that  can  be  used  as  benchmarking. 
First,  GPRA  has  three  components:  mission  statement, 
general  goals  and  objectives,  and  approaches  and  strategies 


95  Robert  B.  Denhardt,  "Strategic  Planning  in  State  and  Local 
Government,"  in  Performance  Based  Budgeting,  ed.  Miller,  Gerald  J.,  W. 
Bartley  Hildreth,  Jack  Rabin.  Anonymous  (Colorado:  Westview  Press, 
2001),  233-244. 

96  ibid. 
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to  achieve  the  General  Goals  and  Ob j ectives . 9^  Second,  the 
Balanced  Scorecard  approach  has  seven  components:  mission, 
vision,  core  values,  goals,  strategy,  strategic  map  and 

ob j  ectives  .  98 

A  mission  statement  is  the  core  purpose  of 
organization.  It  brings  the  agency  into  focus  and  explains 
why  the  agency  exists  and  tells  what  id  does.99A  mission 
statement  should  be  simple,  clear,  inspire  and  easy  to 

understand.  It  clarifies  the  purpose  of  the  organization 
and  clearly  articulates  it  to  all  stakeholders.  In 
designing  the  mission  statements,  several  key  questions  as 
following  have  to  be  answered: 

•  Is  the  mission  results  oriented,  and  does  it 
fulfill  a  public  need?  If  not,  how  could  the 
mission  better  focus  on  results? 

•  Is  the  mission  based  on  statute,  and  if  so, 

does  it  cover  all  relevant  statutes? 

•  Are  parts  of  the  agency' s  functions  or 

activities  not  covered  in  the  mission 
statement?  Why? 

•  Are  there  developments  (e.g.  in  technology  or 
competition)  that  suggest  the  mission  and 
corresponding  legislation  need  to  be  revised  or 
updated? 

•  Is  the  agency's  mission  similar  to  those  of 
other  agencies  and  if  so,  has  coordination 
occurred?  Does  unwarranted  duplication  of 
missions  existed? 

•  How  is  the  agency' s  mission  differentiated  from 
those  of  other  agencies  with  similar  missions? 

Are  there  unique  agency  characteristics  that 


9?  United  States.  General  Accounting  Office,  Agencies'  Strategic  Plans 
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give  it  an  advantage  in  fulfilling  its  mission, 
such  as  location  of  field  offices  or  staff 
expertise?  lOO 

Values  are  the  time  less  principles  that  guide  an 
organization  and  they  represent  the  deeply  held  beliefs 
within  the  organization  and  are  demonstrated  through  the 
day  to  day  behaviors  of  all  employees .  loi  Values  must 
support  the  mission  and  help  achieve  organizational 
objectives.  Patrick  Lencionii02  suggests  organizational 
values  should  be  "aggressively  authentic".  Authentic  in 
this  context  means  developing  values  that  are  consistent 
with  your  organizational  objectives,  not  writing  something 
that  would  be  well  suited  for  the  inside  of  a  greeting 
card.  Johnson  and  Johnson  CEO  Ralph  Larsen  stated  that: 

" . the  core  values  embodied  in  our  credo  might  be 

a  competitive  advantage,  but  that  is  not  why 
would  have  them.  We  have  them  because  they  define 
for  us  what  we  stand  for,  and  we  would  hold  them 
even  if  they  became  a  competitive  disadvantage  in 
certain  situations" . 203 

A  vision  statement  provides  a  word  picture  of  what  the 
organization  intends  ultimately  to  become  in  the  future  and 
while  mission  statements  are  often  abstract,  the  vision 
should  contain  as  concrete  of  the  desired  state  as  possible 
ad  provide  the  basis  for  formulating  strategies  and 
objectives.  i04  Peter  Senge  has  observed:  "Vision  translates 
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mission  into  truly  meaningful  intended  results-and  guides 


the  allocation  of  time,  energy,  and  resources .  In  my 
experience ,  it  is  only  through  a  compelling  vision  that  a 
deep  sense  of  purpose  comes  alive" .  i05 

According  to  Niven,  to  be  effective,  vision  statements 
should  be  concise,  balanced  external  and  internal  elements, 
appealing  to  all  stakeholders,  consistent  with  mission  and 
values,  verifiable,  feasible  and  inspirational.  He 
continues  by  providing  ten  questions  to  develop  vision 
statement : 

•  How  would  the  world  be  improved  or  changed  if 
we  were  successful  in  achieving  our  purpose? 

•  What  are  the  most  important  services  that  we 
should  continue  to  provide,  change,  or  begin  to 
offer  in  the  next  three  years? 

•  What  staffing  and  benefits  changes  do  we  need 
to  implement  to  better  achieve  our  purpose? 

•  How  will  our  elected  officials  or  board  of 
directors  assist  us  in  achieving  our  purpose? 

•  What  resource  development  (funding)  changes  to 
influence  to  better  achieve  our  purpose? 

•  What  facilities  and  technology  changes  to 
implement  to  better  achieve  our  purpose? 

•  What  infrastructure,  systems,  or  communication 
changes  do  we  need  to  implement  to  better 
achieve  our  purpose? 

•  How  could  we  more  effectively  or  efficiently 
provide  our  service?  If  you  could  only  make 
three  changes  that  would  significantly  impact 
our  ability  to  provide  quality  services  to  our 
clients/customers,  what  should  these  be? 

•  What  makes  us  unique? 

•  What  do  our  clients/customers  consider  most 
important  in  our  provision  of  services?  What  do 
our  clients/customers  need  form  us? 


l*-*?  Patrick  M.  Lencioni,  "Make  Your  Values  Mean  Something,"  Harvard 
Business  Review,  July  2002,  pp. 113-117.  In  Paul  R.  Niven,  Balanced 
Scorecard  Step-by-Step  For  Government  and  Nonprofit  Agencies,  (New 
Jersey:  John  Wiley  and  SOns,  Inc,  2003),  3005. 

68 


General  goals  and  objectives  or  strategic  goals 
explain  what  results  are  expected  form  the  agency's  major 
functions  and  when  to  expect  those  results.  i06  Goals  and 
Objectives  should  be  concise,  realistic,  outcome-oriented 
and  must  be  stated  clearly  in  communicating  what  is  be 
achieved  and  measured  or  assessed.  In  public  sector,  the 
goals  and  objectives  should  be  linked  to  government  policy 
decisions  to  create  the  basis  for  performance  reporting. 
Several  key  questions  that  should  be  addressed  in  designing 
goals  and  objectives  are: 

•  Do  the  goals  cover  the  major  functions  and 

operations  of  the  agency?  If  not,  what 

functions  and  operations  are  missing?  Are  the 
goals  logically  related  to  the  mission? 

•  Are  the  goals  results  oriented,  such  as  to 

reduce  crime  or  have  fewer  workplace  accidents? 

Or,  are  they  focused  more  on  outputs,  such  as 
inspecting  more  workplaces?  If  so,  why? 

•  If  the  goals  are  not  expressed  in  a 

quantitative  or  measurable  form,  are  they 
expressed  in  a  manner  allow  the  agency  and 

congress  assessing  whether  goals  are  achieved? 

•  Are  all  the  agency' s  goals  and  priorities 

consistent  with  congress  goals  and  priorities? 

When  difference  exist.  Why  do  they  exist,  and 

can  they  be  resolved? 

•  Do  the  agency' s  goals  appear  similar  to  the 

goals  in  plans  of  other  agencies  that 

performing  related  activities?  If  so,  are  these 
sets  of  goals  complementary  or  duplicative? 

•  Are  the  goals  targeted  at  results  over  which 

the  agency  has  a  reasonable  degree  of  influence 
(may  not  apply  to  all  agencies)  ?i07 
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Strategies  describe  the 


means 


to  achieve  the 


objectives.  It  represents  the  broad  priorities  adopted  by 
an  organization  in  recognition  of  its  operating  environment 
and  in  pursuit  of  its  mission. 108  gaO  states  that  strategies 
help  aligning  an  agency's  activities,  core  processes,  and 
resources  to  support  achievement  of  the  agency' s  strategic 
goals;  mission  and  strategies  should  also  outline  how  the 
agency  will  communicate  strategic  goals  throughout  the 
organization  and  hold  managers  and  staff  accountable  for 
achieving  these  goals.  109  Some  key  questions  to  be  answered 
in  designing  strategy  are: 

•  How  are  the  goals  to  be  achieved?  Are  the 
strategies  logically  linked  to  the  goals  and 
the  day-to-day  activities  of  the  managers  and 
staff?  Are  they  consistent  with  historical 
resource  trends? 

•  What  steps  will  the  agency  take  to  align 
activities,  core  process,  workforce,  and  other 
resources  to  support  mission  related  outcomes? 

•  What  are  the  required  resources,  such  as  human, 
capital,  and  information?  Are  new  regulations, 
flexibilities,  user  fees,  or  legislation 
required? 

•  What  steps  is  the  agency  taking  to  ensure  that 
managers  have  the  authority  they  need  to 
achieve  results?  Are  these  strategies  to  hold 
managers  accountable  for  results?  Are  there  any 
strategies  that  focus  on  providing  incentives 


for  managers 
goals  ? 

and  other 

staff  to 

achieve 

the 

•  Do  managers 

have  the 

knowledge. 

skills 

and 

abilities  to 

implement 

PBB?  If 

not. 

what 

strategies  are  needed  to  develop  the  necessary 
capacity? 


108  Niven,  Balanced  Scorecard  Step-by-Step  For  Government  and  Nonprofit 
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•  Are  technological  advances  necessary  to 
successfully  execute  the  strategies?  If  so,  how 
likely  are  those  advances? 

•  What,  if  any,  alternative  strategies  were 
considered? 

•  Are  there  programs  or  activities  that  need  to 
be  eliminated.  Created,  or  restructured  to 
achieve  the  goals?iio 

In  performance  budgeting  framework,  McGill  states  that 
performance  budgeting  includes  both  a  strategic  framework 
and  the  mechanics  of  resource  allocation  in  relation  to 
performance .  Ill  The  framework  of  strategic  planning  will 
direct  agency  in  creating  specific  targets,  performance 
measures,  and  mechanism  of  resource  allocation  in 
quantifiable  data  to  provide  meaningful  performance 
information  about  program  outcomes. 

These  strategic  contexts  will  help  decision  makers  in 
budgeting  process  to  rely  not  only  based  on  qualitative 
performance  information  but  also  based  on  quantitative 
performance  information.  Here,  it  can  be  noted  that 
strategic  planning  management  plays  an  important  role  in 
PBB .  A  performance  budget  is  one  which  presents  the 
purposes  and  objectives  for  which  funds  are  required,  the 
cost  of  the  programs  proposed  for  achieving  those 
objectives,  and  quantitative  data  measuring  the 
accomplishment  and  work  performed  under  each  program. 112 
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The  important  role  of  strategic  management  also 
provides  an  idea  that  PBB  will  only  fit  in  the  organization 
in  long  term  organization  goals.  In  practice,  it  happens 
because  changes  to  the  budget  process  are  interconnected 
and  include  a  shift  from  the  traditional  annual  budget 
cycle  to  multi  year  frameworks  as  the  basis  for  the  annual 
budget  process.  Performance  based  budgeting  can  function 
somewhat  like  strategic  budgeting  since  it  will  match 
within  organization,  the  mission,  goals  and  objectives,  and 
strategies.  Appropriate  performance  measures  that 
accurately  reflect  agency  efforts  an  accomplishment  are 
essential  for  a  successful  planning  and  budgeting 
process .  ns 

In  the  short  term,  performance  based  budgeting  is  not 
likely  to  have  any  short-run  effect  on  the  way  resources 
are  allocated.  The  reason  for  that  is  because  to  the 
success  of  government  policy  can  not  be  judged  in  short 
run.  There  might  be  delay  between  outputs  and  outcomes,  or 
it  may  not  be  clear  how  much  of  a  change  in  an  outcome  can 
be  attributed  to  a  program  or  policy.  The  design  of 
performance  indicators  and  establishing  policy  objectives 
requires  a  considerable  amount  of  lengthy  consensus 
building  and  performance  indicators  must  remain  stable  over 
time  to  permit  internal  assessment  of  trends  and  to  make 
the  comparative  judgments  needed  to  assess  progress, 

However,  in  practice,  the  underlying  assumptions  that 
PBB  will  fit  in  long  term  is  hard  to  be  accomplished 
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because  changes  in  policies  and  administration.  Schmidtlein 
observes  that: 

. these  conditions  commonly  do  not  exist  because 

government  policy  objectives  typically  are 
ambiguous  and  subject  to  frequent  shifts.  Another 
issues  related  to  this  assumptions  is  that  as  key 
officials  come  and  go  and  political  parties  with 
differing  values  come  to  power,  budgets  are  not 
followed  exactly  as  prescribed  when  they  were 
enacted  because  new  political  deal, 
circumstances,  understanding  arise  for  tactical 
and  partisan  advantage  as  well  as  for  substantive 
reasons . 

D .  PERFORMANCE  MEASURES 

Performance  budgeting  assumes  that  future  budget 
allocations  will  be  influenced  by  performance-informed 
decisions  (which  presupposes  a  targeted  strategic  framework 
from  which  to  make  such  decisions)  it  involves 
developing  indicators  of  institutional  performance  and 
estimating  the  resources  required  to  maintain  or  achieve 
selected  levels  of  performance .  Performance  indicators 
has  to  be  clear  enunciated  so  that  performance  information 
can  help  managers  to  understand  the  performance  of 
organization,  and  to  set  clear  targets  so  that  people 
understand  the  expected  level  of  performance. 

According  to  Kouzmin,  there  has  been  evolution  of 
determining  the  performance  measurement.  Whereas  in  the 
1980s  the  focus  was  on  the  "three  Es",  economy,  efficiency, 
and  effectiveness,  in  the  1990s  attention  has  shifted  to 
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quality  and  consumer  satisfaction .  ns  The  quality  and 
consumer  satisfaction,  which  emphasis  the  efficiency  and 
effectiveness  of  allocating  resources,  has  left  out  the 
economy  perspective.  Especially  in  developed  countries, 
economy  perspective  is  assumed  as  anti-thesis  for  having  a 
good  quality  because  lowest  prices  are  assumed  as  bad 
quality . 

However,  in  accordance  with  the  evolution  of 
performance  measurement  as  mentioned  above  which  according 
to  government  worldwide  is  intended  to  enhance  the  quality 
of  delivery  of  services,  people  are  still  questioning  the 
adequacy  of  government  service  quality.  Although  the  cost 
of  services  is  rising  due  to  lack  of  economy  perspective, 
the  quality  is  inappropriate.  Government  accountability  is 
still  big  question  that  can  be  answered  by  the  government. 
Therefore,  excluding  the  economy  perspective  in  determining 
the  cost  of  services  is  not  the  answer. 

Performance  measures  will  provide  information  on  what 
traditional  "incremental"  budgets  or  financial  statements 
concerned  with  the  results  of  the  services  delivered  by  the 
government.  The  task  of  measuring  the  performance  in  public 
sector  is  complex  and  difficult  due  to  its  intangible 
outputs,  quality  variations,  and  complex  environment.  The 
process  of  government  to  measure  and  judge  its  performance 
by  determining  how  effectively  and  efficiently  taxpayer 
resources  are  being  used  for  the  delivery  of  services  and 
the  administration  of  programs  is  critical  to  enhance 
operational  accountability  of  governments. 
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121 . 


74 


Performance  measures  can  be  classified  into  five 


categories  outputs,  outcomes,  economy,  efficiency  and 
effectiveness.  Outputs  measure  gauge  the  amount  of  services 
delivered  to  citizens  and  allow  one  to  examine  what 
services  and  the  quantities  of  services  an  entity  is 
providing .  119  It  refers  to  the  internal  activities  of  a 
program  (i.e.,  the  products  and  services  delivered)  and 
describes  the  level  of  activity  that  will  be  provided  over 
a  period  of  time,  including  a  description  of  the 
characteristics  (e.g.,  timeliness)  established  as  standards 
for  the  activity.  120 

Outcomes  are  what  an  organization  is  trying  to 
achieve.  It  is  the  impact  sought  or  expected  by  government 
in  a  given  policy  arena,  which  has  focus  on  change  and 
consequences:  what  effect  government  can  have  on  the 

community,  economy  and/or  national  interest.  121  Outcomes 
are  important  to  the  effectiveness  of  programs/activities 
but  may  be  more  difficult  to  measure  and  assess  than  the 
inputs  and  outputs  and  it  will  often  be  influenced  by 
external  factors  and  may  require  long-term  rather  than 
short-term  assessment . 122  Outcome  measures  are  the  ultimate 
results  that  benefit  the  public.  One  of  the  biggest 
problems  in  designing  performance  measures  is  how  to  define 
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the  outputs  and  outcomes  in  an  organization.  Sometimes  it 
is  hard  to  differ  between  the  two  for  public  managers  due 
to  its  intangible  characteristics.  An  example  of  output  and 
outcomes  measures  is  as  shown  here:  123 


Table  6 :  Example  of  Out] 

puts  and  Outcomes  measures 

Outputs 

Outcomes 

Number  of  housing  units 
rehabilitated . 

Increases  in  equity  (property 
value)  of  rehabilitated  houses 
for  low-income  families  as  a 
result  of  targeted  assistance. 

Number  of  businesses 
assisted  through  loans  and 
training . 

Percent  of  businesses  that 
remain  viable  3  years  after 
assistance . 

Number  of  people  served  by 
water/sewer  projects. 

Increased  percent  of  people 
with  access  to  clean  drinking 
water . 

Number  of  acres  of 
agricultural  lands  with 
conservation  plans. 

Percent  improvement  in  soil 
quality;  dollars  saved  in 
flood  mitigation. 

In  establishing  the  outcome  measures,  in  some  cases, 
it  is  hard  for  agencies  and  departments  to  design  the 
outcome  measures.  In  this  matter,  agencies  can  address  this 
problem  by  developing  intermediate  outcomes,  that  is, 
partial  outcomes  which  are  achieved  within  a  shorter  time 
frame  and  this  would  enable  the  better  identification  of 
target  groups  and  provide  a  more  appropriate  basis  for  the 
development  output. 124  Following  example  can  be  use  as  a 
drilling  down  to  get  to  the  right  outcome  measure  for  a  job 

training  program:  125 
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Table  7 :  Performance  Measures  on  Job  Training  Program 


o  Dollars  appropriated  to  the  program 
o  Number  and  size  of  grants 

Why  do  these  matters?  What  do  they  buy? 

Inputs : 

Funding  (Federal  and 
perhaps  State  and 
local ) 

o  Number  of  classes  attended  by  program 

Outputs : 

participants 

Products  (e . g . , 

o  Number  of  people  trained 

classes  taught. 

Why  do  these  matters?  What  result  do  they 

service  delivered. 

produce ? 

participants  serviced) 

o  Number  of  people  with  useful  skills 

Intermediate  outcomes: 

o  Number  of  people  who  get  a  job  after 

(e.g.,  new  knowledge. 

leaving  the  program 

increased  skills. 

Why  do  these  matters?  Is  this  the  result  the 
public  is  seeking? 

changed  behavior) 

o  Number  of  program  participants  who  remain 
employed  for  a  specified  time  and  increase 
their  earnings 

o  Number  of  people  who  are  self-sufficient 

Program  outcome 

Societal  outcome 

An  economy  measure  looks  at  the  costs  of  acquiring  the 
inputs  which  compare  the  resources  available  to  be 
allocated  and  those  used  as  input  for  performing  an 
activity.  Economy,  which  occurs  where  equal-quality 
resources  are  acquired  at  lower  prices,  is  minimizing  the 
cost  of  resources  used  for  an  activity,  having  regard  to 
the  appropriate  quality  and  focus  on  the  cost  of  the  inputs 
and  processes .  126  Economy  not  only  means  to  have  the  lowest 
price  but  also  to  have  cost  reduction  in  allocating  the 
resources  to  save  the  taxpayer  money. 

An  efficiency  measure  looks  at  whether  the  maximum 
outputs  for  the  inputs  that  go  into  the  process  have  been 
achieved.  Efficiency  is  'the  relationship  between  the 
output,  in  terms  of  goods,  services  or  other  results,  and 
the  resources  used  to  produce  them  and  it  exists  where  the 

126  Auditing  Standards  Committee,  International  Organization  of 
Supreme  Audit  Institutions,  Auditing  Standards, 
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use  of  financial,  human,  physical  and  information  resources 
is  such  that  output  is  maximized  for  any  given  set  of 
resource  inputs,  or  input  is  minimized  for  any  given 
quantity  and  quality  of  output. "127  Sound  efficiency 
measures  capture  skillfulness  in  executing  programs, 
implementing  activities,  and  achieving  results,  while 
avoiding  wasted  resources,  effort,  time,  and/or  money. 128 

According  to  0MB,  the  best  efficiency  measures  capture 
improvements  in  program  outcomes  for  a  given  level  of 
resource  used,  for  example,  a  program  that  has  an  outcome 
goal  of  "reduced  energy  consumption"  may  have  an  efficiency 
measure  that  shows  the  value  of  energy  saved  in  relation  to 
program  costs.  However,  it  may  be  difficult  to  express 
efficiency  measures  in  terms  of  outcomes,  because  in  some 
cases,  acceptable  efficiency  measures  could  focus  on  how  to 
administer  the  program  better.  129  0MB  continues  by  stating 
that  meaningful  efficiency  measures  consider  the  benefit  to 
the  customer  and  serve  as  indicators  of  how  well  the 
program  performs,  for  example,  reducing  processing  time 
means  little  if  error  rates  increase.  A  balanced  approach 
is  required  to  enhance  the  performance  of  both  variables  in 
pursuit  of  excellence  to  customers.  In  these  instances,  one 
measure  (e.g.,  increase  in  customer  satisfaction)  may  be 
used  in  conjunction  with  another  complementary  measure 
(e.g.,  reduction  in  processing  time) .  130 


127  Auditing  Standards  Committee,  International  Organization  of 
Supreme  Audit  Institutions,  Auditing  Standards, 

128  Office  of  Management  and  Budget,  0MB' s  Program  Assessment  Rating 
Tool:  FY  2006  PART  Instructions, 

129  ibid. 

130  ibid. 
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Effectiveness  measures  gauge  the  quality  and 
productivity  in  the  delivery  of  a  service  to  citizens  and 
it  has  characteristics  including  citizen's  evaluations, 
timeliness,  readiness,  conditions  and  accuracy.  It  looks 
at  whether  the  outputs  of  the  program  lead  to  the  desired 
outcomes.  According  to  INTOSAI,  132  effectiveness  is  the 
extent  to  which  objectives  are  achieved  and  the 
relationship  between  the  intended  impact  and  the  actual 
impact  of  an  activity  and  it  addresses  the  issue  of  whether 
the  program/activity  has  achieved  its  objectives.  INTOSAI 
also  states  that  when  focusing  on  effectiveness,  it  is 
important  to  distinguish  between  the  immediate  outputs  (or 
products)  and  the  ultimate  impacts  (or  outcomes) . 

In  designing  performance  measures  in  accordance  with 
agencies  goals  and  objectives  as  well  as  their  outcomes,  it 
is  possible  that  among  agencies  have  the  same  goals, 
objectives  and  outcomes.  Crosscutting  is  a  performance 
measurement  method  to  evaluate  interdependent  government 
programs  which  have  similar  purposes,  goals,  and  outcomes. 
This  could  be  an  internal  crosscut,  programs  within  the 
same  agency,  or  external  crosscut,  programs  across  multiple 
agencies.  Crosscutting  would  identify  exemplary  goals  and 
practices,  common  measures  of  performance,  possible 
tradeoffs  in  management  and  budget  decisions,  and 
opportunities  for  better  coordination  among  programs .  133 


131  Donovan,  Performance  Measurement:  Connecting  Strategy,  Operations 
and  Actions , 

132  Auditing  Standards  Committee,  International  Organization  of 
Supreme  Audit  Institutions,  Auditing  Standards, 

133  Office  of  Management  and  Budget,  0MB' s  Program  Assessment  Rating 
Tool:  FY  2006  PART  Instructions, 
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Crosscutting  can  be  designed  by  having  a  joint  working 
among  departments  and  agencies  which  have  similar  outcome. 
In  choosing  the  most  appropriate  measurement  method  for 
encouraging  joint  working,  factors  to  be  considered 
include:  (1)  the  number  of  departments  and  other 
stakeholders  involved;  (2)  the  degree  of  stakeholder 
interpretation  of  national  objectives  needed  for  cost 
effective  pursuit  of  objectives;  (3)  the  priority  accorded 
the  objectives;  and  (4)  the  costs  and  burdens  of  given 
approaches  to  setting  targets  and  monitoring  progress.  134 

In  United  States,  crosscutting  is  designed  in  PART 
assessment.  Unlike  a  combined  PART,  a  crosscut  would  still 
examine  programs  in  individual  PARTs .  By  having  the 
crosscut,  individual  assessment  of  programs  in  individual 
PART  and  the  recognition  of  the  distinctive  features  among 
programs  still  can  be  evaluated.  However,  according  to  0MB, 
the  following  common  themes  could  be  used  to  compare  the 
programs  in  a  meaningful  crosscut: 

•  What  is  the  target  population  for  each  program? 

•  What  products  and  services  are  provided  (common 
output  measures  as  appropriate) ? 

•  How  well  products  and  services  are  provided 
(common  output  efficiency  measures  as 
appropriate) ?  What  impact  have  the  programs 
achieved  (common  outcome  measures  and  common 
outcome  efficiency  measures  as  appropriate) ?i35 


134  The  Comptroller  and  Auditor  General,  Measuring  the  Performance  of 
Government  Departments , 

135  Office  of  Management  and  Budget,  0MB' s  Program  Assessment  Rating 
Tool:  FY  2006  PART  Instructions, 
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After  the  performance  measures  are  gauged,  it  is 
important  to  compare  the  results  against  established 
standards.  Standards  can  be  factors  such  as  the  level  and 
quality  of  client  services  and  are  set  with  the  aim  of 
defining  the  appropriate  level  of  performance  expected  to 
be  delivered.  Standards  or  benchmarks  could  be  set  by 
professional  organizations,  the  performance  of  other 
government  entities,  composite  indices  of  select  government 
entities,  or  pre-set  targets  or  previous  performance  of 
entity .  136 

However,  a  comprehensive  set  of  benchmarked  results  in 
the  public  sector  have  not  been  embraced  by  the  government 


it  self  or  professionals. 

An 

alternative  to 

replace 

the 

benchmarks 

or  standards  is 

to 

use 

performance 

targets 

and 

baselines 

just  like  United 

States 

has  in  GPRA. 

Baselines 

are  the  starting  point  from  which  gains  are  measured  and 
targets  are  set  and  the  baseline  year  shows  actual  program 
performance  or  prior  condition  for  the  given  measure  in  a 
specified  prior  year.  137  Targets  refer  to  improved  levels 
of  performance  needed  to  achieve  the  stated  goals  and  they 
must  be  ambitious  (i.e.,  set  at  a  level  that  promotes 
continued  improvement)  and  achievable  given  program 
characteristics;  each  target  should  be  quantifiable  and 
must  have  a  timeframe  (e.g.,  year(s)  in  which  the  target 
level  is  to  be  achieved)  .  In  most  instances,  these  targets 
should  be  quantifiable  .  i38 


136  Bourne,  Mike  and  John  Mills,  "Designing,  Implementation  and 
Updating  Performance  Measurements  Systems,"  International  Journal  of 
Operations  &  Production  Management  20  (2000)  :  754-771. 

137  Office  of  Management  and  Budget,  0MB’ s  Program  Assessment  Rating 
Tool:  FY  2006  PART  Instructions, 

138  ibid. 
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E. 


LINKING  PERFORMANCE  TO  BUDGET 


Finally,  the  process  that  must  be  established  is 

linking  performance  to  budget.  The  performance  measurement 
is  tied  directly  to  the  budget  process  comprehensively  and 
elaborates  it  to  evaluate  organization's  performance  in 

terms  of  outcome,  output,  and  impact.  It  will  give  a 

meaningful  indication  of  how  the  dollars  are  expected  to 

turn  into  results  and  it  can  be  distinguished  from  an 

Object  Class  budget  in  a  fundamental  way  because  while  in 
the  object  class  budget  what  each  dollar  will  be  spent 
would  be  showed,  in  the  performance  budgeting,  inherently, 
what  each  dollar  will  accomplish  also  would  be  showed.  i39 

In  integrating  performance  into  the  budget, 
performance  measures  should  capture  the  essence  of  the 

policy  objective  and  the  contribution  of  program  activities 
to  the  outcome.  Performance  information  could  help 

policymakers  address  a  number  of  questions,  such  as  whether 
programs  are  contributing  to  their  stated  goals,  well 
coordinated  with  related  initiatives  at  the  federal  level 
or  elsewhere,  and  targeted  to  those  most  in  need  of 

services  or  benefits.  It  can  also  provide  information  on 
what  outcomes  are  being  achieved,  whether  resource 

investments  have  benefits  that  exceed  their  costs,  and 
whether  program  managers  have  the  requisite  capacities  to 
achieve  promised  results. i^o 

In  this  stage,  at  least  there  three  important  point 
that  has  to  be  considered  in  how  to  link  performance  to 
budget  resources.  First  is  the  budget  alignment  in  which 

139  "Performance  Based  Budgeting:  Overview,"  [cited  2004],  Available 
from  http : / / www . j  ohn-mercer . com/ pbb . htm 

140  United  States.  General  Accounting  Office,  Performance  Budgeting; 
Current  Development  and  Future  Prospects,  24. 
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structural  and  format  changes  to  congressional  budget 
justifications,  and  in  some  cases,  appropriations  accounts 
to  better  align  resources  with  programs  and  performance. 
The  need  to  change  the  account  structure  is  important 
because  some  performance  goals  cut  across  multiple  program 
activities  and  budget  accounts  or  a  single  program  can 
consist  of  multiple  accounts.  Budget  restructuring  efforts 
have  sought  to  help  reframe  budget  choices  and  to  focus 
decisions  more  on  the  expected  results  of  budget  resources 
and  less  on  inputs  or  line  items. i^i 

The  second  aspect  is  costing  method.  There  are  two 
methods  that  can  be  used:  full  cost  and  activity  based 
costing  (ABC)  .  Full  cost  is  the  way  in  which  changes  to 
certain  budget  resources  are  distributed  or  measured  to 
better  reflect  where  and  when  resources  are  consumed.  0MB 
defined  full  cost  as  the  sum  of  all  budget  resources  used 
by  an  agency  to  achieve  program  outputs.  1^2  The  budget 
resource  can  include  not  only  traditional  elements  of 
costs,  such  as  salaries  and  expenses,  procurement  of  goods 
and  services,  grants,  and  transfers  but  also  the  cost  of 
all  support  services  and  goods  used  and  provided  centrally 
which  can  be  including  accruing  retiree  pension  and  health 
benefits.  Budgeting  for  the  full  cost  of  resources  where 
they  are  used,  making  budget  program  and  activity  lines 
more  parallel  with  outputs,  and,  where  useful,  improving 
alignment  of  budget  process. 


141  United  States.  General  Accounting  Office,  Performance  Budgeting; 
Efforst  to  Restructure  Budgets  to  Better  Align  Resources  with 
Performance  (Washington  D.C.:  U.S.  General  Accounting  Office,  2005), 
183. 

142  OMB,  Circular  No.  A-11  (July  2002),  p.  221-5  in  ibid. 

143  ibid. 
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Full  cost  is  generally  viewed  as  including  both  direct 
costs,  costs  that  can  be  specifically  identified  with  a 
cost  object,  such  as  an  output,  materials,  and  supplies 
used  in  work,  office  space,  and  equipment  and  facilities 
that  are  used  exclusively  to  produce  the  output,  1^4  and 
indirect  cost,  costs  of  resources  that  are  jointly  or 
commonly  used  to  produce  two  or  more  types  of  outputs  but 
are  not  specifically  identifiable  with  any  of  the  outputs 
such  as  general  administrative  services,  general  research 
and  technical  support,  security,  rent,  and  operations  and 
maintenance  costs  for  building,  equipment  and  utilities, 

Activity  Based  Costing  (ABC)  is  used  to  identify, 
describe,  assign  costs  to  and  report  on  agency  operations; 
it  identifies  opportunities  to  improve  business  process 
effectiveness  and  efficiency  by  determining  the  "true"  cost 
of  a  product  or  service,  abc  principles  are  used  to  focus 
management  attention  on  the  total  cost  to  produce  a  product 
or  service  and  as  the  basis  for  full  cost  recovery.  The 
process  ABC,  started  by  determining  the  cost  activity  pools 
based  on  defined  activities,  assigning  the  costs  the 
activity  cost  pools,  calculating  the  activity  rates  and 
finally  assign  the  costs  to  cost  objects  using  the  activity 
rates  and  activity  measures,  will  provide  users  with  better 
understanding  of  the  cost  information  for  strategic 
decisions . 


144  OMB,  Circular  No.  A-11  (July  2002),  p.  221-5  in  ibid. 

145  OMB,  Circular  No.  A-11  (July  2002),  p.  221-5  in  ibid. 

146  "Activity-Based  Costing:  Accounting  for  Operational  Readiness,"  in 
The  Learning  Source  OSD  Comptroller  iCenter  [database  online] .  [cited 
2005].  Available  from  http://www.ditc.mil/comptroller. 
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ABC  can  become  the  basis  for  establishing  performance 
measures;  targeting  cross-functional  for  improvement 
analysis;  determining  the  gap  in  performance  that  may 
require  reengineering;  benchmarking  process  improvement  to 
identify  proven  practices  within  organization  and  with 
external  competitors  and  other  organizations;  and 
developing  performance  based  budgeting  plans . 147 

The  third  important  aspect  is  designing  the  assessment 
tool.  The  Performance  Assessment  Rating  Tool  (PART)  in  GPRA 
is  maybe  one  of  the  best  practices  that  can  be  used  in  this 
matter.  PART  is  basically  generated  from  the  strategic 
goals  and  developed  to  be  specific  operational  performance 
goals.  Performance  goals  are  expressed  in  a  measurable  and 
attainable  objective  to  be  matched  with  actual  achievement 
in  terms  of  outputs  of  outcomes. 

According  to  OMB148,  the  PART  requires  two  types  of 
performance  goals.  First,  long-term  performance  goals 
address  performance  that  is  generally  several  years  or  more 
in  the  future  which  includes  two  basic  types  of  long-term 
goals,  an  annual  performance  goal  in  the  future  (e.g., 
tornado  warning  times  in  2008,  or  unit  costs  of  an  activity 
in  2010) ;  and  the  cumulative  effect  of  annual  activities 
(e.g.,  development  of  an  AIDS  vaccine  by  2010)  .  Second, 
Annual  performance  goals  should  be  stated  in  yearly 
increments  For  example,  for  the  weather  program,  an  annual 
performance  goal  might  include  the  same  performance  measure 
(advance  warning  time),  but  a  less  ambitious  target  (e.g.. 


147  Tracey  G.  Amos,  Cynthia  A.  Paolillo,  and  Denise  A.  Joseph, 

"Enhancing  CFO,  GMRA  &  GPRA  implementation  with  activity  based 
management,"  The  Government  Accountants  Journal  46,  no.  1  (1997)  :  28. 

148  Office  of  Management  and  Budget,  0MB' s  Program  Assessment  Rating 
Tool:  FY  2006  PART  Instructions, 
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15  minutes  average  warning  time  in  2005)  due  to  less 
widespread  use  of  advanced  technologies. 

PART  which  contains  25  questions  (See  Appendix  B  for 
detailed  PART  Questions)  is  divided  into  four  sections  with 
specific  weight  for  each  section  to  create  the  numerical 
scores  in  the  four  management  categories  as  following: 


Table  8:  Overview  of  Sections  of  PART  Questions 


Section 

Description 

Weight 

I .  Program  Purpose  & 
Design 

To  assess  whether 

•  The  purpose  is  clear,  and  The 
program  design  makes  sense 

20% 

To  assess  whether  the  agency 
sets  valid  programmatic 

II.  Strategic  Planning  10% 

•Annual  goals  and  Long  term 

goals 

To  rate  agency  management  of  the 
program  including 

III.  Program  Management  20-5 

•Financial  oversight,  and 

Program  improvement  efforts 

To  rate  program  performance  on 

goals  reviewed  in 

^  50% 

•The  strategic  planning  section, 

and  Through  other  evaluations 

Source:  From  GAO  Report,  GAO-04-174,  p.  9.149 


To  aid  standardization,  the  questions  that  comprise 
the  PART  are  generally  written  in  a  Yes/No  format.  The 
part's  instructions  (see  Appendix  B  for  detailed 
Instructions  of  the  Program  Assessment  Rating  Tool)  make  a 
clear  point  that  to  be  awarded  a  yes  answer;  the  program 
should  have  a  high  standard  of  performance  with  a  high 
level  of  confidence.  However,  according  to  GAQi^o^  the 


IV.  Program  Results/ 
Accountability 


149  United  States.  General  Accounting  Office,  Results-Oriented 
Government:  GPRA  Has  Established  a  Solid  Foundatuon  for  Achieving 
Greater  Results,  270. 

150  United  States.  General  Accounting  Office,  Performance  budgeting 
[electronic  resource]  :  observations  on  the  use  of  OMB’s  program 
assessment  rating  tool  for  the  fiscal  year  2004  budget, 
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format  of  yes  and  no  response  has  resulted  in 
oversimplified  answers  to  some  questions.  Some  parties  felt 
that  the  format  did  not  adequately  distinguish  between  the 
performances  of  various  programs.  To  solve  the  problem,  in 
spring  2002,  0MB  revised  the  PART  to  include  four  responses 
by  adding  "small  extent"  and  "large  extent"  to  the  original 
two  choices  "yes"  and  "no")  .  It  can  be  seen  from  standard 
of  a  yes  in  fiscal  year  2006  PART  instruction: 

The  PART  holds  programs  to  high  standards.  Simple 
adequacy  or  compliance  with  the  letter  of  the  law 
is  not  enough.  Rather,  a  program  must  show  it  is 
achieving  its  purpose  and  that  it  is  well 
managed.  The  PART  requires  a  high  level  of 
evidence  to  justify  a  Yes  response.  Sections  I 
through  III  are  scored  in  a  Yes/No  format.  In 
Section  IV,  a  four-level  scale  (Yes,  Large 
Extent,  Small  Extent,  and  No)  permits  answers  to 
reflect  partial  achievement  of  goals  and  evidence 
of  results.  Answers  must  be  based  on  the  most 
recent  credible  evidence,  i^i 


To  convert  the  PART  scores  into  qualitative  ratings, 
0MB  uses  the  following  scoring  bands:  effective  (85-100), 
moderately  effective  (70-84),  adequate  (50-69),  and 
ineffective  (0-49) .  If  the  measures  lack  baselines  and 
performance  data  and  the  program  does  not  have  performance 
measures  that  have  been  agreed-upon  by  0MB,  a  rating  of 
Results  Not  Demonstrated  (RND)  is  given.  Gilmour  and  Lewis 
state  that  the  last  category  is  not  intended  to  reflect 
well  or  poorly  on  a  program.  Rather,  the  "the  results  not 
demonstrated"  grade  reflects  0MB' s  judgment  that  the 


151  Office  of  Management  and  Budget,  0MB’ s  Program  Assessment  Rating 
Tool:  FY  2006  PART  Instructions, 


87 


existing  measures  for  a  program  do  not  allow  an  assessment 
of  the  program's  ef  f  ectiveness  .  1^2 

The  outputs  of  PART  then  will  be  transferred  into  a 
scorecard.  The  0MB ' s  Executive  Branch  Management  Scorecard 
contains  six  "Core  Criteria" .  The  highest  rating  of  "green" 
a  federal  agency  must  satisfy  all  six  criteria.  A  rating  of 
"yellow"  is  awarded  if  the  agency  meets  six  similar 
criteria  that  establish  a  lower  level  of  achievement. 
However,  if  any  one  of  six  defined  "Conditions"  is  found  to 
exist,  the  agency  will  automatically  be  given  the  lowest 
rating  of  "red."  The  detail  of  the  scorecard  core  criteria 
can  be  seen  in  Appendix  C. 

By  having  performance  measures  linked  to  budget,  if 
PBB  could  provide  what  government  programs  have  good 
results  or  poor  results,  how  the  government  will  be 
encouraged?  Will  government  be  rewarded  or  punished?  Under 
performance  budgeting,  people  should  not  expect  that  good 
results  will  always  be  rewarded  through  the  budget  process 
while  poor  results  will  always  have  negative  funding 
implications .  153  However,  decisions  on  funding  levels  may  be 
intended  either  to  reward  high  achievement  or  to  penalize 
inadequate  achievement .  154  CAO  continues  by  observing  that 
viewing  performance  budgeting  as  a  mechanistic  arrangement- 
a  specific  level  of  performance  in  exchange  for  a  certain 

152  John  Gilmour  B.  and  David  Lewis  E.,  "Assessing  Performance 
Assessment  for  Budgeting:  The  Influence  of  Politics,  Performance,  and 
Program  Size  in  FY  2005,"  Journal  of  Public  Administration  Research  and 
Theory  (2004) 

153  United  States.  General  Accounting  Office,  Performance  Budgeting; 
Current  Development  and  Future  Prospects,  24. 

154  Schmidtlein,  Assumptions  Underlying  Performance-Based  Budgeting, 
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amount  of  funding-or  in  punitive  terms-produce  results  or 
risk  funding  reductions-is  not  useful.  Such  mechanistic 
relationships  cannot  be  sustained.  Rather  than  increase 
accountability  these  approaches  might  instead  devalue  the 
process  by  favoring  managers  who  meet  expectations  by 

aiming  low.  155 

McNab  and  Melese  observe  that  if  the  GPRA  is  to  create 
an  explicit  linkage  between  budget  appropriation  (and 

obligations)  and  the  outcomes  generated  by  public 
expenditure,  it  must  assist  in  the  creation  of  an 

institutional  environment  that  rewards  efficiency, 
transparency,  and  the  prompt,  concise,  and  accurate 
reporting  of  costs,  outputs,  and  outcomes.  i56  in  the  absence 
of  such  an  environment,  departments  may  respond  to  the 
current  incentive  structure  by  "gaming"  their  performance 
reports  to  present  their  activities  in  terms  designed  to 
maximize  their  budget.  157 

GAO  claims  that  PBB  can  play  an  important  role  in 

creating  incentives  in  a  pay-f or-perf ormance  system,  for 
example,  once  a  sufficient  level  of  result  is  achieved, 
managerial  bonuses  might  be  linked  to  maintaining  that 
level  or  service,  while  decreasing  the  program  unit  costs; 
this  creates  a  particularly  strong  incentive  fro  managers 
not  to  spend  every  program  dollar  where  it  is  not  necessary 

155  United  States.  General  Accounting  Office,  Performance  Budgeting; 
Current  Development  and  Future  Prospects,  24. 

156  McNab  and  Melese,  Implementing  the  GPRA:  Examining  the  Prospects 
for  Performance  Budgeting  in  the  Federal  Government,  73-95. 

157  ibid. 


89 


for  achieving  the  desired  result.  158  This  is  also  the  reason 
why  public  managers  shifting  their  attention  from 
traditional  "incremental"  line  item  budgeting  to 
performance  based  budgeting. 

F .  CHAPTER  SUMMARY 

Performance  based  budgeting  can  be  used  as  a 
supporting  tool  for  the  line  item  budgeting  since  PBB 
provide  the  performance  information  that  can  not  be 
provided  by  line  item  budgeting.  Performance  Based 
Budgeting  basically  is  a  set  of  processes  translating  the 
strategic  planning  framework  into  performance  measures  in 
terms  of  input,  output,  and  outcome  to  be  tied  into 
resource  allocation  in  budgeting  process.  Three  principles 
that  must  be  established  in  PBB  are  strategic  planning, 
performance  measurement  and  linking  performance  to  budget. 

Albeit  strategic  planning  is  still  new  for  the  public 
sector,  several  reasons  such  as  to  give  clarity  and 
direction  to  the  organization,  to  choose  from  among 
competing  goals  and  activities,  to  cope  with  expected 
shifts  in  the  environment,  and  to  bring  together  the 
thoughts  and  ideas  of  all  participants  in  the  work  of  the 
organization  has  shifted  the  public  sector  manager  to 
utilize  it  in  their  organization.  In  designing  the 
components  of  strategic  planning,  there  are  two  approaches 
that  can  be  used  as  benchmarking.  First,  GPRA  has  three 
components:  mission  statement,  general  goals  and 


"Congressional  Testimony;  Statement  of  John  Mercer  before  the 
Subcommittee  on  Government  Efficiency,  Financial  Management,  and 
Intergovernmental  Relations  U.S.  House  Representative,  June  19,  2001; 
The  Results  Act:  Has  it  Met  Congressional  Expectations?"  [cited  2004], 
Available  from  http://www.john-mercer.com/library/jn  testimony.pdf. 
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objectives,  and  approaches  and  strategies  to  achieve  the 
General  Goals  and  Objectives.  Second,  the  Balanced 
Scorecard  approach  has  seven  components:  mission,  vision, 
core  values,  goals,  strategy,  strategic  map  and  objectives. 

In  designing  the  performance  measurement  system, 
although  there  has  been  evolution  of  determining  the 
performance  measures  in  which  the  focus  has  been  shifted 
from  in  the  1980s  on  the  "three  Es",  economy,  efficiency, 
and  effectiveness,  to  in  the  1990s  on  quality  and  consumer 
satisfaction,  the  factor  of  the  wealthy  of  a  country  can  be 
used  in  determining  which  measures  that  will  be  taken. 
While  developed  countries  which  have  enormous  resources  to 
be  allocated  for  government  budget  might  use  the  quality 
and  consumer  satisfaction  which  emphasis  the  efficiency  and 
effectiveness  assessment,  developing  countries  which  have 
limited  resources  might  not  leave  the  economy  perspective 
to  obtain  cost  reduction  in  government  budget. 

In  linking  performance  to  budget,  the  performance 
measures  not  only  capture  the  essence  of  the  policy 
objective  and  the  contribution  of  program  activities  to  the 
outcome  but  also  provide  information  on  what  outcomes  are 
being  achieved,  whether  resource  investments  have  benefits 
that  exceed  their  costs,  and  whether  program  managers  have 
the  requisite  capacities  to  achieve  promised  results.  Three 
important  aspects  that  have  to  be  considered  in  how  to  link 
performance  to  budget  resources  are:  budget  account  changes 
to  better  align  resources  with  programs  and  performance; 
full  cost  or  ABC  to  better  reflect  where  and  when  resources 
are  consumed;  and  the  assessment  tool  in  which.  The 
Performance  Assessment  Rating  Tool  (PART)  in  GPRA  is  can  be 
used  as  the  best  practice. 
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IV.  A  MODEL  FOR  INDONESIAN  DOD 


A.  INTRODUCTION 

This  chapter  provides  a  model  of  performance  based 
budgeting  for  Indonesian  DOD/TNI  by  incorporating  the 
knowledge  and  information  from  the  Chapter  II  and  Chapter 
III.  Section  B  presents  a  short  overview  of  budgeting 
process  in  Indonesia.  Section  C  discusses  the  progress  of 
the  implementation  of  Performance  Based  Budgeting  in 
Indonesia  as  well  as  several  comparisons  between  the 
current  progress  and  United  States  Experiences  in  GPRA. 
Section  D  provides  a  cascade  model  of  performance  based 
budgeting  process  for  Indonesian  DOD/TNI.  A  summary  is 
provided  at  the  end  of  the  chapter  in  Section  E 

B .  OVERVIEW 

The  1945  Constitution  of  the  Republic  of  Indonesia 
recognizes  six  state  institutions:  (1)  The  People's 

Deliberative  Assembly  or  Ma jells  Permusyawaratan  Rakyat 
(MPR) ;  (2)  The  President;  (3)  The  Supreme  Advisory  Council 

or  Dewan  Pertimbangan  Agung  (DPA  ) ;  (4)  The  House  of 

People's  Representatives  or  Dewan  Perwakilan  Rakyat  (DPR) ; 
(4)  The  Audit  Board  of  the  Republic  of  Indonesia  or  Badan 
Pemeriksa  Keuangan  (BPK) ;  (5)  The  Supreme  Court  or  Mahkamah 

Agung  (MA) . 

The  hierarchical  position  of  the  six  institutions  is  as 
following : 


93 


Figure  1:  Hierarchical  Position  of  Six  Supreme  Institutions 

in  Indonesia 


Notes : 

1.  MPR:  The  People's  Deliberative  Assembly  or  Majelis 
Permusyawaratan  Rakyat; 

2.  DPA:  The  Supreme  Advisory  Council  or  Dewan 

Pertimbangan  Agung; 

3.  DPR:  The  House  of  People's  Representatives  or  Dewan 
Perwakilan  Rakyat; 

4.  BPK:  The  Audit  Board  of  the  Republic  of  Indonesia  or 
Badan  Pemeriksa  Keuangan; 

5.  MA:  The  Supreme  Court  or  Mahkamah  Agung. 

From  the  above  figure,  it  can  be  seen  that  the  highest 
constitutional  body  is  the  MPR  which  is  selected  every  five 
years.  Half  of  the  MPR  members  are  assigned  as  member  of  DPR 
which  is  constitutionally  has  legislative  authority.  This 
condition  also  gives  strong  power  to  DPR.  However,  it  can  be 
dissolved  by  President.  If  the  President  exceeds  his 
authority  the  DPR  can  convene  the  MPR  to  bring  the  President 
to  account.  The  President  as  the  principal  executor  of  state 
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government  is  not  responsible  to  DPR  but  to  MPR  except  for 
the  law  enactment  and  state  budget  approval  in  which 
President  needs  DPR  approval.  BPK  is  responsible  for 
auditing  the  government  accountability  in  using  the  state 
funds  and  delivering  the  report  to  the  DPR.  Ministries  and 
Heads  of  Agencies  are  appointed  by  the  President  and  are 
responsible  to  the  President.  For  government  structure,  the 
Indonesian  Government  has  a  three  tier  government 
structures;  state,  regional,  and  local. 

Indonesian  budgeting  system  has  evolved  dramatically 
for  the  past  two  years,  especially  with  enormous  pressures 
from  investors  to  better  manage  financial  management  and 
enhance  government  accountability.  Lessons  learned  from  the 
fragile  system  before  the  monetary  and  economic  crisis  in 
1998  have  provoked  the  stakeholders  to  demand  changes 
towards  how  the  government  runs  its  business. 

In  the  past,  the  financial  management  system  was  linked 
to  the  Indische  Compatabiliteits  Wet  (ICW)  inherited  from 
Ducth  Colonialism,  and  further  amended  from  time  to  time  and 
called  the  Indonesian  Financial  Administration  Act. 
Realizing  the  inadequacy  of  the  ICW,  gradually  the 
government  has  enacted  several  laws  related  to  financial 
matters.  The  important  law  that  has  become  the  starting 
point  of  the  willingness  to  change  is  the  State  Finance  Law 
No.  17/2003.  Before  the  enactment  of  this  law,  the  budgeting 
system  was  the  traditional  line  item  budgeting  system.  The 
state  budget  was  prepared  on  the  basis  of  the  Five  Year 
Development  Plans  (REPELITA)  in  accordance  with  the  General 
Guidelines  of  State  Policy  (GBHN)  .  GBHN  was  a  long  term 
national  development  plan  for  a  twenty-five  year  period  and 
was  updated  every  five  years. 
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The  budget  appropriation  was  prepared  on  an  annual 
basis.  The  budget  estimates  were  prepared  annually  and 
submitted  to  the  DPR  for  approval.  Any  policy  and  provision 
changes  on  the  approved  budget  need  DPR  approval.  The  budget 
bill,  having  approved  by  the  DPR  and  signed  by  the 
President,  comprised  of  state  revenues  and  state 
expenditures.  While  state  revenues  consisted  of  two 
classifications.  First  was  routine  revenues,  revenues 
internally  generated  from  taxes  revenues  and  non  taxes 
revenue  (PNBP) ;  second  was  development  revenues,  revenues 
received  in  foreign  loans.  State  expenditures  also  consisted 
of  two  classifications:  routine  expenditures  and  development 
expenditures.  They  were  allocated  into  sectors  and  sub 
sectors . 

C.  PERFORMANCE  BASED  BUDGETING  IN  INDONESIA 

Before  the  government  reform  which  was  initiated  after 
the  1998  crisis,  there  was  no  performance  evaluation  tool 
provided  by  the  government  to  assess  the  execution  of  its 
tasks  and  activities.  Government  had  a  strong  power  over 
all  other  supreme  institutions  in  the  country.  The  only 
performance  evaluation  was  DPS,  a  performance  evaluation 
for  civil  servant.  However,  this  tool  was  still 
inappropriate  and  consisted  of  many  unclear  and  biased 
measures . 

After  the  1998  crisis,  feeling  pressure  from  investor 
countries,  a  series  of  action  was  taken.  The  first  effort 
was  by  launching  the  President  Instruction  No.  7,  1999  that 
required  departments,  agencies  and  regional  and  local 
government  to  prepare  the  Accountability  and  Performance 
Reports,  AKIP  report  (LAKIP-Laporan  Akuntabilitas  Instansi 
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Pemerintah)  .  However, 

it 

has 

a  serious 

flaw  in  that  the 

agency  or  department 

is 

free 

to  assign 

a  weight  to  each 

activity  performed 

and 

the 

danger  is 

that  the  agency 

attempt  to  engineer  the  weight  in  order  to  search  the 
"best"  result,  regardless  the  improvement  and  furthermore, 
this  has  been  perceived  as  a  useless  exercise  since  there 
is  no  follow-up  or  utilization  of  the  LAKIP  data. 159 

The  second  effort  is  the  involvement  of  investor 
countries  through  Non  Governmental  Organization  and 
consultants  is  to  assist  local  governments  in  preparing  a 
new  result  oriented  budget,  performance  budgeting.  This 
effort  has  been  taken  in  accordance  with  the 
decentralization  policy  which  was  initiated  in  1999.  It  is 
clearly  stated  in  the  8th  paragraph  of  Government 
Regulation  (PP)  No.  105/2000  that  local  government  should 
follow  performance  budgeting  in  preparing  the  annual 
budget.  However,  the  biggest  challenge  to  implement  the 
performance  budgeting  in  local  government  is  lack  of 
understanding  of  the  concept  of  performance  budgeting. 
Several  other  issues  faced  by  local  government  are  lack  of 
general  guidelines  for  local  planning;  delayed  release  of 
new  guidelines  for  local  budgeting  and  financial 
management;  vague  expenditure  planning;  and  lack  of 
qualified  local  planning  and  budgeting  of f icials . 160 

For  central  government,  by  the  enactment  of  two  laws, 
the  Law  No.  17/2003  and  the  National  Development  Planning 
System  Law  No  25/2003  gradually  the  government  transforms 
how  they  run  business.  The  National  Development  Planning 

159  Ahmadi  H.  Ringoringo,  Performance  Budgeting;  The  Impact  of  Pubic 
Expenditure  Policy  to  Local  Government , Anonymous  ,  Jakartaed.  Project 
497-0357/104-000,  USAID-funded  Partnership  for  Economic  Growth,  2002) 

160  ibid. 
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System  Law  No.  25,  2003  was  enacted  to  shift  the  national 
planning  from  GBHN  system  to  the  new  planning  system  so 
called  PROPENAS  (National  Planning  Program) .  By  this  new 
law,  the  planning  system  will  initiate  strategic  planning  by 
incorporating  a  twenty-year  period  of  long  term  planning, 
divided  into  four  medium  term  planning  (five  years)  and 
short  term  annual  planning.  The  PROPENAS  provides  directions 
for  development  strategy  at  all  levels  of  government,  which 
determines  sectoral  allocations  of  development  spending  for 
central  and  regional  levels.  Strategic  development  plans  of 
central  line  ministries  and  agencies  (RENSTRA)  are  produced 
based  on  PROPENAS,  accompanied  by  the  annual  national 
development  plan  (REPETA)  defining  priorities  in  the 
national  development  budget. 

The  State  Finance  Law  No.  23  2003  provides  significant 
changes  for  state,  regional  and  local  budgeting  systems. 
Ministry  of  Finance  has  cooperated  with  other  related 
agencies  and  consultants  to  prepare  the  manual  and 
additional  regulation  to  cover  the  implementation  of 
performance  budgeting.  It  is  expected  that  the  full 
implementation  will  be  started  in  FY  2006.  There  are  three 
important  changes  for  the  budgeting  process.  The  first 
important  aspect  in  this  law  is  the  requirement  for  state 
and  local  governments  to  implement  results  oriented 
budgeting  by  integrating  the  performance  accountability 
system  into  the  budgeting  system.  The  second  important 
aspect  is  the  requirement  to  unify  the  budget 
classification,  called  unified  budget.  This  change  will 
eliminate  the  previous  budget  classification  of  routine  and 
development  budgets  which  created  redundancy,  inefficiency 
and  even  fraud  in  departments  and  agencies.  The  third 
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important  aspect  is  Medium  Term  Expenditure  Framework  (MTEF) 
to  enhance  the  inter-correlation  between  the  planning  and 
budgeting  process  and  to  obtain  more  rational  and  strategic 
resources  allocation.  It  is  also  mentioned  that  in  future 
year  government  will  provide  Government  Accounting  Standard 
so  that  government  financial  statements  can  be  more 
transparent  and  accountable. 

In  the  State  Budget  for  FY2005,  the  government  has  made 
some  progress  and  adjustment  in  accordance  with  the  Law  no. 
23/2004.  The  important  move  is  that  government  has 
reclassified  the  budget  accounts.  The  purposes  of  the  budget 
reclassification  are  (1)  to  avoid  budget  duplication  due  to 
lack  of  policy  to  distinguish  between  operational  routine 
activities  and  development  project  in  the  past,  (2)  to 
ease  the  preparation  of  performance  based  budgeting  and  to 
explain  relationship  between  output/outcome,  (3)  to  provide 
objective  and  proportional  descriptions  about  government 
financial  activities,  and  (4)  to  increase  government 
financial  statistical  credibility . i6i  The  central  government 
expenditures  are  divided  into  eleven  "function"  categories: 
(1)  public  service,  (2)  defense,  (3)  Safety  (4)  economy  (5) 
environmental,  (6)  housing  and  public  facilities,  (7) 
healthcare,  (8)  tourism  and  culture  (9)  religion,  (10) 
education,  and  (11)  social  protection . i62 

The  unified  budget  principle  in  this  budget  reform  is 
very  important  in  ensuring  that  investment  and  recurrent 
operational  expenditures  simultaneously  are  considered  in 
budget  decision  making.  It  will  also  assist  in  preparing  the 


161  Indonesian  Ministry  of  Finance,  State  Budget  Plan  FY2005, 
(Jakarta:  Indonesian  Ministry  of  Finance,  2005),  162. 

162  ibid. 
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performance  based  budgeting  manual  that  is  currently  being 
prepared  by  the  government.  Some  basic  changes  on  budget 
format  due  to  the  implementation  of  unified  budget  and 
budget  accounts  reclassification  are  as  following: 

Table  9 :  Changes  on  Budget  Format  Due  to  the  Implementation 


of  Unified  Budget  and  Budget  Accounts  Reclassification 


Old  Format 

New  Format 

>  Expenditures  Classification 
•  Dual  budgeting 

•Expenditures  consisted  of  six 
types  of  expenditures 

including  Development 

Expenditures 

>  Expenditures  Classification 
•Unified  budgeting 

•Central  Government 

Expenditures  consist  of  eight 
type  of  expenditures 

^  Organization  Classifications 

were  not  included  in  Government 
Finance  Notes  and  State  Budget 
Law 

^  Organization  Classifications 

are  included  in  Government 

Finance  Notes  and  State  Budget 
Law 

^  Sector  Classification 

•Consisted  of  20  sectors  and  50 

sub  sectors 

•  Programs  were  sub 

classification  of  sector  on 
routine  and  development  budget 

^  Sector  Classification 

•Consists  of  11  functions  and 
79  sub  functions 

•Programs  are  classified  based 
on  departments  and  agencies 
and  then  compiled  based  on 
functions 

^  Budget  Allocation  was  based  on 
sector,  sub  sector,  and  program 

^  Budget  Allocation  is  based  on 
government  departments  and 

agencies 

Source:  From  Indonesian  State  Budget  Plan  FY2005,  p.  96163 


The  detailed  format  changes  especially  on  state 
expenditures  based  on  economic  classification  can  be 
described  in  the  following  table: 


163  ibid. 
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Table  10:  Conversion  State  Expenditures  Based  on 
Expenditures  Types 


Old  Format 

New  Format 

A.  State  Revenues  and  Donation 

A.  State  Revenues  and 

I . Internal  Revenues 

Donation 

1 . Tax  Revenues 

I .  Internal  Revenues 

2 . Non  Tax  Revenues 

1 . Tax  Revenues 

II.  Foreign  Aid  Grant 

2 . Non  Tax  Revenues 

B.  State  Expenditures 

II.  Foreign  Aid  Grant 

I . Central  Government 

B.  State  Expenditures 

Expenditures 

I . Central  Government 

1 . Routine  Expenditures 

Expenditures 

a. Salary  Expenditures 

1 . Salary  Expenditures 

b . Goods  Expenditures 

2 . Goods  Expenditures 

c. Interest  Expense 

3. Capital  Expenditures 

d. Subsidies 

4 . Interest  Expense 

e. Other  Routine 

5 . Subsidies 

expenditures 

6 . Foreign  Aid 

2 . Development  Expenditures 

Expenditures 

7 . Social  Assistance 

II.  Local  Government 

8 . Other  expenditures 

Expenditures 

II.  Local  Government 
Expenditures 

C.  Primary  Balance 

D.  Budget  Surplus/deficit 

C .  Primary  Balance 

E.  Financing  (e.g.  Foreign 

D. Budget  Surplus/deficit 

Loan  and  loan  from 

E .  Financing 

National  Bank) 

Source:  From  Indonesian  State  Budget  Plan  FY2005,  p.  9tl64 


Budget  classification  is  based  on  function  and  programs 
instead  of  on  sector/sub-sector  in  the  old  format. 
Function/sub-function  is  not  the  basis  of  budget  allocation. 
Budget  allocation  is  based  on  the  proposed  programs  from 
departments  and  agencies  and  then  the  programs  are 
classified  by  their  functions  and  sub-functions.  Hence,  the 
detail  of  the  expenditure  budget  is  the  compilation  of  the 
budget  from  departments  and  agencies. 

By  comparing  United  States  experience  with  its  GPRA  and 
the  current  progress  of  performance  budgeting  implementation 


164  ibid. 
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in  Indonesia  several  lessons  learned  can  be  noted.  First, 
the  GPRA  is  the  law  and  has  support  from  both  President  and 
Congress.  The  position  as  a  law  has  provided  strong  power 
for  department  and  agencies  to  implement  it.  Otherwise,  the 
content  of  GPRA  has  appropriately  provided  information  on 
how  to  implement  the  law,  ranging  from  the  strategic 
planning,  performance  measures,  and  linking  performance  to 
budget  or  budget  performance  and  integration  as  well  as  the 
performance  plans  and  reports. 

The  Law  No.  17.2003  does  not  provide  detail  like  GPRA 
has  since  it  only  provides  the  several  statements  on  the 
requirement  to  implement  performance  based  budgeting.  For 
the  detail  about  performance  based  budgeting,  currently,  the 
government  still  discuss  about  this  matter.  To  be  able  to 
emulate  the  success  of  GPRA,  the  Indonesian  government  has 
to  provide  the  new  law  on  the  detail  explanation  and  process 
of  performance  budgeting.  By  enacting  it  as  a  law  which 
means  it  will  need  DPR  approve,  minimally  it  will  take  DPR 
attention  and  government  can  cooperate  with  the  DPR  from  the 
earlier  process  of  performance  budgeting  so  that  both 
parties  agree  with  the  content  of  the  law  and  support  the 
implementation . 

Second,  since  being  enacted  in  1993,  the  implementation 
of  GPRA  was  initiated  with  several  pilot  projects  and 
divided  into  two  categories,  the  strategic  planning, 
performance  measures  and  the  managerial  flexibility  waiver. 
The  implementation  was  also  being  delayed  due  to  several 
problems  mentioned  in  Chapter  II.  Learning  from  those 
experiences,  it  would  be  appropriate  for  Indonesian 
Government  not  to  speed  up  the  implementation  in  2006  in 
whole  departments  and  agencies.  It  will  be  better  if  this 
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project  is  started  with  several  pilot  projects  in  all 
departments  and  agencies  so  that  the  budget  efficiency  and 
quick  response  towards  any  deviation  and  changes  of  the 
standards  and  procedures  could  be  attained. 

Finally,  the  most  important  thing  related  to 
performance  measurement  and  assessment  is  standards  and 
benchmarking.  The  lack  of  performance  standards  and 
benchmarking  available  for  the  public  sector  has  been  a 
problem  for  budget  audit  and  evaluation  stage  in  comparing 
between  performance  plans  and  actual  realization.  The  method 
used  in  GPRA  of  having  annual  performance  plans  and  annual 
performance  report  to  be  compared  in  performance  evaluation 
stages  can  be  emulated.  This  will  also  save  department  and 
agencies  time  in  preparing  the  budget  instead  of  paying  much 
attention  in  designing  and  finding  appropriate  standards  and 
benchmarking . 

D.  PBB  MODEL  FOR  DOD 

1 .  Overview 

It  is  an  ironic  that  Indonesia,  a  country  with  a  vast 
geographical  area,  a  high  degree  of  vulnerability  to  both 
internal  and  external  threats  and  a  strategic  location  in 
the  Asia-Pacific  region  currently  has  not  more  than  50% 
military  readiness  capabilities  on  main  weaponry  system.  For 
example,  from  200  jet  fighters,  comprising  F-16  Fighting 
Falcons,  A4-Sky  Hawks,  Hawk  200s  and  F-5s  and  about  75 
percent  of  the  fighters,  however,  are  unable  to  fly  due  to 
lack  of  spare  parts.  Several  jet  fighters  are  already 
obsolete  and  in  need  of  replacement.  Some  A-4  Sky  Hawks,  for 
example,  were  made  in  1980  but  they  are  still  being  flown 
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today,  roughly  13  years  after  they  should  have  been  grounded 
in  1990. 

It  is  absolutely  understandable  that  Indonesia  needs  to 
acquire  more  sophisticated  weapons  and  use  them  in  defending 
its  territorial  integrity  and  protecting  the  country 
adequately.  This  condition  has  been  exacerbated  by  having 
the  military  embargo  imposed  by  the  United  States  following 
the  violence  in  East  Timor  in  1999.  The  embargo  has 
prevented  Indonesian  National  Armed  Force  (TNI)  from  getting 
spare  parts,  ammunition,  and  other  military  hardware  from 
United  States.  Albeit,  several  talks  between  Indonesia  and 
US  have  showed  a  promising  result,  however,  officially  U.S. 
still  has  not  lifted  the  embargo. 

This  has  made  it  difficult  to  provide  funding  for  the 
real  needs  of  TNI.  Defense  structure  is  very  expensive  so 
that  it  should  be  planned  carefully  with  sufficient 
considerations  towards  the  availability  and  sustainability 
of  state  funds  as  well  as  internal  and  external  political 
environment.  The  last  effort  by  government  in  procuring 
four  Sukhoi's  through  counter  trade  system  with  Russia  was 
one  the  efforts  to  enhance  the  capabilities  by  using  off- 
budget  payment  and  exchanging  forty  commodities  for  several 
deliveries  in  the  future. 

The  condition  above  coupled  with  an  increasing  public 
attention  to  enhance  the  capabilities  from  increasing 
external  threats  as  well  as  TNI's  willingness  to  reform 
itself  into  a  new  paradigm;  there  has  been  a  dramatic 
budget  increase  for  the  last  three  years  on  defense  budget. 
For  example,  from  only  approximately  US$800  million 
(FY2002$)  in  2002  which  was  less  than  4  percent  of  the 
government  budget,  for  2005,  the  military  budget  is 
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approved  for  approximately  US$  2.4  billion  (FY2005$)  which 
is  almost  10%  of  government  budget. 

The  budget  increase  does  not  mean  that  TNI/DOD  can 
spend  it  without  having  adequate  control.  To  create 
transparency  as  part  of  civilian  control  over  military, 
especially  in  budget  control,  all  military  disbursements 
should  be  performed  in  accordance  with  available  laws  and 
regulations  by  planning,  budgeting,  managing  and  monitoring 
them  appropriately.  The  DPR  as  representative  of  society 
must  be  fully  involved  in  the  process  of  budgeting.  A  clear 
justification  of  confidentiality  towards  national  security 
should  be  formulated  sufficiently  to  avoid  misusing  of  the 
funds  in  particular  activities.  The  public  has  a  right  to 
know  what  really  happens  in  the  country  and  to  obtain 
sufficient  information  about  problems. 

To  fulfill  the  demand  in  transforming  into  the  new 
paradigm  and  to  provide  the  strategic  framework  as  well  as 
to  increase  the  accountability  and  present  the  transparency 
to  the  public,  DOD  has  also  issued  the  Defense  White  Paper 
which  provides  the  strategic  framework  of  how  DOD/TNI 
perceives  the  threats,  defines  its  capabilities,  and 
articulates  strategic  policy  for  Indonesian  defense. 
Before  2003,  where  the  only  strategic  framework  document 
was  what  was  available  in  GBHN,  DOD/DNI  did  not  have  any 
document  such  as  National  Military  Strategy  in  the  U.S. 
that  was  publicly  available  to  provide  the  strategic 
framework  of  how  DOD  should  run  its  business. 

The  military  budget  system  is  much  different  from 
other  civil  government  departments  and  agencies. 
Internally,  in  budget  formulation  and  costing,  DOD  has 
already  implemented  the  Planning,  Programming,  and 
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Budgeting  System  (PPBS) .  After  introduction  in  the  1970' s, 
the  system  has  been  adjusted  gradually  in  accordance  with 
the  development  of  financial  system  in  the  country.  The 
outputs  of  the  PBBS  system  are  coordinated  into  one  budget 
request  that  will  be  submitted  to  the  Ministry  of  Finance. 

For  program  and  budget  management,  DOD/TNI  has  three 
basic  principles:  bottom  up,  top  down,  and  one  gate  policy. 
The  bottom  up  policy  means  that  every  unit  hierarchically 
proposes  the  business  need  plans  based  on  proposal  as  well 
as  its  explanation  including  the  detail  calculations  and 
arguments  to  support  the  budget  plan.  The  top  down  policy 
means  top  management  determines  the  lower  level  budget 
based  on  priority,  corporate  policies,  and  resources 
capabilities.  One  gate  policy  means  every  function, 
planning,  programming,  executing,  and  controlling  in 
organization  is  conducted  separately  by  function  in  the 
organization . 

In  its  response  to  the  implementation  the  unified 
budget,  DOD  has  also  changed  its  budget  classification  into 
programs  for  the  FY  2005.  The  new  Defense  Budget  program 
structure  with  functional  approach  can  be  seen  as  following: 
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Table  11:  Indonesian  DOD  Budget  FY  2005  Based  on  Functions 


(in  thousands.  Rupiah) 


Code 

Program 

Amount 

02.01 

Country  Defense 

02.01.01 

Integrated  Defense  Development 

2,150,010,040 

02.01.02 

Land  Defense  Development 

9, 052, 604, 211 

02.01.03 

Sea  Defense  Development 

3, 187,952,828 

02.01.04 

Air  Defense  Development 

2,377, 112, 942 

02.01.05 

Strengthening  Nation  Unity  and  Sovereignty 

25,759, 920 

02.02 

Defense  Supports 

02.02.01 

Defense  System  and  Strategy  Development 

52,587,019 

02.02.02 

Defense  Supporting  Potency  Development 

5, 016, 172,184 

02.02.03 

Defense  Industries  Development 

19, 314, 956 

02.03 

Overseas  Military  Cooperation 

02.03.01 

International  Military  Cooperation 

40,789,490 

02.04 

Defense  Research  and  Development 

02.04.01 

Defense  Research  and  Development 

28,756, 549 

02.04.02 

National  Defense  Development 

15,500, 000 

02.05 

Others  Defense 

02.05.01 

TNI  Social  Operation 

26, 569, 085 

Total 

21,977,629,824 

Source:  From  Indonesian  State  Budget  Plan  FY2005,  p.lSS^^S 


Based  on  the  table  above,  although  the  budget  has  been 
adjusted  to  program  structure,  from  the  program  title,  it 
can  be  seen  that  the  programs  are  too  general.  If  the 
DOD/TNI  is  willing  to  use  this  program  classification  in  the 
performance  based  budgeting,  it  will  result  in  the  programs 
that  are  too  wide,  consisting  of  many  overlapping 
performance  measures.  Moreover  it  will  make  the  process  of 
setting  and  evaluating  the  performance  measures  difficult. 
It  will  be  better  if  the  program  is  created  more 
specifically  so  that  the  performance  measures  can  be 
directly  tied  to  the  program. 

The  above  configuration  will  be  different  if  we  compare 
them  with  the  United  States  DOD  Programs  directed  to 
specific  strategies:  such  as  Air  Combat  Program,  Air  Force 
Aircraft  Operations,  Air  Force  Depot  Maintenance,  Airlift 

165  ibid. 
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Program,  Army  Land  Forces  Operations,  Basic  Research, 
Chemical  Demilitarization,  Communications  Infrastructure, 
Defense  Health,  Depot  Maintenance  -  Naval  Aviation,  Depot 
Maintenance  -  Ship,  DOD  Applied  Research  Program,  DOD  Small 
Business  Innovation  Research  Technology,  Energy  Conservation 
Improvement,  Housing,  Military  Force  Management,  Missile 
Defense,  Navy  Ship  Operations,  Navy/Marine  Corps  Air 
Operations,  Recruiting,  Shipbuilding,  and  Facilities 
Sustainment,  Restoration,  and  Modernization. 

For  strategic  planning,  because  DOD  and  Indonesian 
National  Armed  Forces  (TNI)  are  separated  and  have  equal 
position  under  the  President,  there  is  no  such  a  unitary 
strategic  planning.  Ideally  and  officially,  both 
organizations  have  to  have  the  same  mission,  visions,  goals 
and  objectives,  and  strategies.  Albeit,  financially  and 
administratively,  TNI  is  willing  to  merge  with  the  DOD, 
hierarchically,  TNI  is  still  reluctant  to  be  under  the  DOD 
just  like  the  U.S.  has  in  its  DOD.  If  this  condition  would 
be  resolved,  conflicting  of  interests  in  civil  military 
relationships  will  continue  to  escalate  in  the  future 
years.  The  current  TNI  mission,  visions,  and  corporate 
level  goals  are: 
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Table  12:  Indonesian  TNI  Mission,  Vision,  Goals 


Mission  Statement 

"To  build  a  solid  institution  of  Indonesian  Armed 
Forces  (TNI)  and  to  give  priority  to  the 
professionalism  as  the  guardian  of  the  nation 
freedom  and  safety." 

Vision  Statement 

"To  complete  the  internal  reformation  and  put  TNI 
back  as  the  nation  tool  of  defense  by  holding  on  to 
Propenas  2002  -  2004  decision  of  defense  and  safety 
development . " 

Corporate  Level 
Goals 

1.  Restructure  the  TNI  in  accordance  with  new 

paradigm  consistently  through  reposition, 

redefinition  ,  and  reactualization  of  TNI's  role 
to  protect,  maintain  and  defend  the  unitary  state 
of  the  Republic  of  Indonesia  against  internal  and 
external  threats  by  incorporating  the  human 
rights  and  perform  services  in  assisting  the 
development  of  the  country 

2.  Develop  the  nation  defense  and  security  system 
capabilities 

3.  Enhance  the  quality  of  TNI  professionalism, 
enhance  the  force  ratio  of  main  components,  and 
develop  regional  capabilities  supported  with 
appropriate  infrastructure  and  funding. 

4.  Widen  and  enhance  the  quality  of  defense 

bilateral  cooperation  to  maintain  the  stability 
of  regional  safety  and  to  participate  in 

maintaining  the  world  peace. 

Strategies 

1.  Build  the  Force  Structure:  Build  military 

operational  capabilities  to  defend  and  not  for 
defense,  enhancing  management  quality, 

intelligent  capabilities,  territorial  functions 
enhancement,  and  cooperation  with  allies. 

2.  Force  development  through  limited  personnel  and 
equipment  regeneration  and  personnel  management 
with  task  orientation. 

3.  Enhancing  the  quality  Main  Weaponry  System 

through  Research  and  Development  activities  and 
existing  main  weaponry  system  maintenance 

4.  Military  Deployment  is  developed  in  conjunction 

with  defense  policies  and  strategies  with 

geographical  configuration  orientation. 

2 .  The  Cascade  Model  of  Performance  Based  Budgeting 

for  Indonesian  DOD/TNI 

Based  on  the  comparative  study  with  United  States 
experiences  with  GPRA  in  Chapter  II  as  well  as  the 
theoretical  literature  review  in  Chapter  III,  the  framework 
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of  the  cascade  model  of  Performance  Based  Budgeting  for 
Indonesian  DOD/TNI  can  be  described  in  following  figure: 


Figure  2:  Performance  Based  Budgeting  Process 


The  PBB  process  is  started  by  designing  the  strategic 
plan.  Strategic  planning  consists  of  mission,  goals  and 
objectives,  and  strategies  that  provide  strategic  direction 
for  the  organization  and  inspire  the  entire  human  resources 
in  the  DOD/TNI  to  better  manage  and  conduct  all  necessary 
activities  to  achieve  the  mission.  To  be  able  to  design  an 
effective  performance  measurement  system,  DOD/TNI  has  to 
prepare  the  strategic  plan  so  that  it  can  generate  a  sound 
measurement  system. 


no 


The  strategic  plan  has  to  be  well  documented.  The 
example  of  a  strategic  plan  can  be  described  in  the 


following  figure: 

Figure  3 :  Example  of  DOD/TNI  Strategic  Plan 

DOD/TNI  Strategic  Plan 
(FY  2006-2011) 

Mission  Statement:  To  build  a  solid  institution  of  Indonesian 

Armed  Forces  (TNI)  and  to  give  priority  to  the 
professionalism  as  the  guardian  of  the  nation 
freedom  and  safety. 

Vision  Statement:  To  complete  the  internal  reformation  and  put 

TNI  back  as  the  nation  tool  of  defense  by  holding 
on  to  Propenas  2002  -  2004  decision  of  defense  and 
safety  development. 

Strategic  Goal  1 :  Restructure  the  TNI  in  accordance  with  new 

paradigm  consistently  through  reposition, 
redefinition,  and  reactualization  of  TNI's  role 

Strategic  Goal  2 :  Develop  and  maintain  the  nation  defense  and 

security  system  capabilities 

□  Strategic  Objective  2.1.  Fulfill  the  optimum  existing  force 
readiness  level 

•  Strategies  2. 1.1.1.  Enhancing  the  quality  Main  Weaponry 
System  through  Research  and  Development  activities 

•  Strategies  2. 1.1. 2.  Enhancing  the  quality  Main  Weaponry 
System  through  maintaining  existing  main  weaponry  system 
configuration 

□  Strategic  Objective  2.2.  Fulfill  TNI  Personnel  capabilities 
up  to  95  percent 

Strategic  Goal  3:  Enhance  the  quality  of  TNI  professionalism, 

enhance  the  force  ratio  of  main  components,  and 
develop  regional  capabilities  supported  with 
appropriate  infrastructure  and  funding. 

Strategic  Goal  4:  Build  and  enhance  the  quality  of  defense 

bilateral  cooperation  to  maintain  the  stability  of 
regional  safety  and  to  participate  in  maintaining 
the  world  peace. 


the  figure  above,  it  can  be  seen  that  the 
goal  is  more  specifically  defined  by 


From 

strategic 
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two 


strategic  objectives,  one  of  which  is  "Fulfill  the  optimum 
main  weaponry  system  readiness  level."  The  strategic 
performance  goal  is  a  measurable  target  level  to  be 
achieved  by  FY  2011  in  which  it  is  the  measurable  outcome 
that  DOD/TNI  wants  to  achieve.  To  achieve  this  objective, 
two  strategies  are  provided:  (1)  Enhancing  the  quality  Main 
Weaponry  System  through  Research  and  Development  activities 
and  (2)  Enhancing  the  quality  Main  Weaponry  System  through 
maintaining  existing  main  weaponry  system  configuration. 

In  linking  strategic  goals  and  objectives  with 
performance  measures,  in  Eigure  3  below,  the  goal  is 
divided  in  two  types:  Strategic  Performance  Goals  and 
Annual  Performance  Goals.  A  strategic  performance  goal  will 
function  as  a  long  term  measurement  in  medium  term 
framework  while  an  annual  performance  will  function  as  an 
intermediate  outcome  for  specific  fiscal  year  budget.  To 
achieve  the  80%  mission  capable  rates  of  existing  main 
weaponry  system  by  2011,  for  EY  2006,  the  intermediate 
outcome  is  assumed  at  60%  from  2005  baseline  in  the 
percentage  of  aircraft  that  are  capable  of  performing  their 
designated  mission. 

Based  on  the  strategies  in  Eigure  2,  a  series  of 
programs  have  to  be  created  to  achieve  the  goals  and 
objectives.  In  the  figure  3,  the  example  of  program  is  Air 
force  Depot  Maintenance  Program.  The  program  will  have 
three  performance  measures,  an  output  and  two  outcomes.  The 
output  is  the  number  of  the  number  of  maintenance  actions 
performed  in  FY  2006  and  completed  on  schedule,  1250 
actions  in  which  this  the  number  of  maintenance  actions 
that  will  be  covered  in  FY2006  budget.  The  two  outcomes  for 
the  programs  are  first  "Reduce  the  percentage  of 
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maintenance  actions  performance  in  the  depot  that  are 
completed  on  schedule"  and  second  "Maintain  no  more  than 
30%  Organic  Aircraft  Quality  Defect  Rate  (defects  per 
aircraft) . " 

Figure  4 :  Example  of  End  Outcome  and  Intermediate  Outcome 

to  outputs 


strategic  Goal  2 :  Develop  and  maintain  the  defense  and 

security  system  capabilities 

□  Strategic  Objective  2.1.  Fulfill  the  optimum  main  weaponry 
system  readiness  level 

^  Strategic  Performance  Goal  2.3.1.  Maintaining  80% 
mission  capable  rates  of  existing  main  weaponry  system 
by  2011 

O  FY  2006  Annual  Performance  Goal  2. 3. 1.1.  Achieve  60% 
from  2005  baseline  in  the  percentage  of  aircraft  that 
are  capable  of  performing  their  designated  mission. 

D  Air  force  Depot  Maintenance  Program 

•  Program  Measure  2. 3. 1.1  -  PM. A.  Output:  The 
number  of  maintenance  actions  performed  in  FY 
2006  and  completed  on  schedule  is  1250. 

•  Program  Measure  2. 3. 1.1  -  PM.B.  Outcome:  Reduce 
the  percentage  of  maintenance  actions 
performance  in  the  depot  that  are  completed  on 
schedule 

•  Program  Measure  2. 3. 1.2  -  PM.C.  Outcome: 

Maintain  no  more  than  30%  Organic  Aircraft 
Quality  Defect  Rate  (defects  per  aircraft) 


In  budget  preparation  stage,  two  processes  have  to  be 
performed:  budget  formulation  and  costing  in  DOD/TNI  and 
budget  plan  evaluation  in  the  Ministry  of  Finance.  The 
output  of  budget  formulation  and  costing  is  such  an  initial 
budget  request.  Deriving  strategic  planning  and  performance 
measures  in  previous  stages,  in  budget  formulation  and 
costing,  DOD/TNI  have  to  provide  the  scope,  content. 
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performance  and  quality  of  programs  and  activities  proposed 
to  meet  the  organization's  mission,  goals  and  objectives. 

In  this  stage,  the  most  important  tool  that  has  to  be 
applied  is  selecting  the  costing  method.  DOD/TNI  has  to 
choose  which  method  would  be  appropriate  for  their 
business,  between  full  cost  and  ABC.  Probably  most 
government  departments  and  agencies  would  use  the  full  cost 
method  since  it  is  easier  to  align  the  budget  account  from 
previous  format,  however  since  the  DOD/TNI  has  used  unit 
cost  in  designing  the  budget,  ABC  might  be  the  good  choice 
for  the  costing  method. 

In  preparing  a  budget  request  that  will  be  delivered 
to  the  Ministry  of  Finance,  the  budget  request  includes  a 
strategic  plan  which  provides  a  logical  connection  between 
the  strategic  and  performance  context  and  day  to  day  basis 
activities,  performance  information  in  terms  of  output, 
intermediate  outcome,  final  outcome,  efficiency,  and 
effectiveness,  and  cost  information  which  identifies  the 
real  cost  of  providing  services  with  cost  charged  to  the 
appropriate  program.  The  information  on  the  budget  request 
above  can  be  used  to  provide  budget  justification  that  will 
be  negotiated  in  the  Ministry  of  Finance,  to  make  tradeoffs 
among  the  subunits  in  allocating  the  funds  based  on 
availability  and  priorities;  to  measure  productivity  of  the 
subunits,  and  to  determine  overlapping  services  within  the 
DOD/TNI.  In  the  Ministries  of  Finance,  a  performance 
informed  budget  will  be  used  in  analyzing  the  budget 
request  from  TNI/DOD  in  the  context  of  its  performance 
implications  or  outcomes,  to  make  tradeoffs  among  the 
departments  and  agencies,  and  to  provide  budget 
justification  for  submission  to  the  DPR. 


114 


The  budget  approval  stage  involves  two  players, 
government  represented  by  the  Ministry  of  Finance  as  well 
as  DOD/TNI  and  the  DPR.  In  this  stage,  to  obtain  approval 
from  the  DPR  and  to  defend  the  budget  request,  performance 
information  can  help  the  government's  capacity  to  assess 
competing  claims  in  the  budget  by  arming  the  DPR  with 
better  information  on  the  performance  results  for  both 
individual  program  as  well  as  entire  government  budget 
portfolios  in  addressing  performance  outcomes.  Furthermore, 
it  helps  clarify  the  relationship  between  the  budget  and 
the  results,  in  which  the  performance  information  not  only 
shows  the  results  of  a  certain  budget  program  but  also 
indicates  how  those  funds  are  expected  to  generate  those 
results.  To  maximize  control,  the  DPR  could  focus  primarily 
on  expected  results  rather  than  in  the  amount  of  funds 
appropriated.  This  would  assist  agencies  in  developing 
priorities,  since  authorization  of  legislation  involves 
reaching  consensus  between  the  DPR  and  the  Government. 

In  the  budget  execution  stage,  the  important  point  is 
how  to  continuously  update  and  gather  of  performance 
information.  The  information  should  be  reported  at 
frequent,  specified  dates,  with  department  heads  being  held 
responsible  for  both  accurate  data  and  improvements  in 
performance.  The  potential  benefits  for  DOD/TNI  of  using 
performance  information  in  this  stage  are  to  understand  the 
specific  implications  of  the  approved  budget  for 
performance,  to  analyze  the  progress  of  the  programs  by 
comparing  the  targeted  performance  with  the  current 
progress,  to  assist  in  making  contract  decision  and  its 
management  as  well  as  measuring  the  contractor  performance, 
and  to  monitor  the  budget  and  performance  realization  and 
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provide  quick  feedback  for  unintended  and  intended 
deviation  budget  execution. 

For  budget  audit  and  evaluation,  the  performance 
information  can  be  used  to  shift  the  focus  of  audits  and 
evaluations  to  include  performance  questions,  rather  than 
only  financial  compliance  so  that  internal  and  external 
auditors  are  able  to  determine  the  success  or  failure  of  a 
program  as  well  as  the  compliance  with  applicable  law. 
Generally,  the  performance  information  will  help  the  work 
of  auditor  in  conducting  performance  audits.  Without  having 
a  performance  system,  auditors  are  not  able  to  conduct  the 
audits  because  in  performance  audit,  the  organization's 
performance  measurement  system  has  to  be  compared  with 
standards  and  benchmarks.  Specifically,  the  performance 
measure  elements  such  resources,  inputs,  outputs,  outcomes 
would  be  used  to  assess  the  economy  (resources  VS  inputs) , 
efficiency  (inputs  VS  outputs) ,  and  cost  effectiveness 
(resources  VS  outcomes)  and  program  effectiveness  (outputs 
VS  outcomes)  aspects  of  programs  or  organizations. 

E .  CHAPTER  SUMMARY 

For  the  last  few  years,  Indonesia  has  proved  itself  to 
be  a  more  democratic  country  by  incorporating  a  series  of 
reform  to  enhance  government's  accountability.  In  the 
budgeting  system,  the  State  Finance  Law  No.  23  2003  provides 
significant  changes  on  the  state,  regional  and  local 
budgeting  system  through  four  important  changes:  performance 
based  budgeting,  unified  budget.  Medium  Term  Expenditure 
Framework,  and  Government  Accounting  Standard.  Progress  such 
as  budget  account  reclassification  by  implementing  the 
unified  budget  as  well  as  strategic  planning  can  be  found  in 
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regional  and  local 


almost  every  department,  agency, 
government  and  has  proved  that  the  government  has  speeded  up 
the  PBB  implementation. 

In  the  DOD/TNI,  as  an  agency  which  obtains  significant 
budget  increase  for  the  last  5  years,  the  performance  based 
budgeting  has  to  be  well  provided  so  that  public  can  be 
better  informed  on  how  DOD  manages  the  budget.  The 
performance  based  budgeting  model  provided  in  this  Chapter 
can  be  used  as  an  initial  step  to  implement  the  PBB. 

The  PBB  model  starts  with  the  strategic  planning  and 
consists  of  mission,  goals  and  objectives,  and  strategies 
that  provide  strategic  direction  for  the  organization  and 
inspire  the  entire  human  resources  in  the  DOD/TNI  to  better 
manage  and  conduct  all  necessary  activities  to  achieve  the 
mission.  The  performance  measures  that  will  be  used  are 
output,  outcome,  economy,  efficiency,  and  effectiveness.  In 
linking  the  performance  to  budget,  the  budget  request 
including  the  strategic  performance  context,  performance 
information,  and  cost  information  has  to  be  applied  and 
provided  in  the  four  stages  of  budgeting,  budget 
preparation,  budget  approval,  budget  execution,  and  budget 
evaluation  and  audit. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

Performance  based  budgeting  has  been  acknowledged  as 
the  new  proven  successful  method  budget  reform  in  the 
developed  countries  such  as  United  States,  New  Zealand, 
Sweden  and  United  Kingdom.  The  willingness  of  Indonesian's 
government  to  adopt  performance  based  budgeting  would 
provide  an  umbrella  for  government  performance  evaluation 
tool  in  enhancing  public  sector  accountability.  However 
this  effort  should  result  from  the  inner  will  to  change 
instead  of  to  fulfill  demand  and  pressure  from  the  donor 
countries . 

To  permeate  a  performance  perspective  into  budget 
information  and  decision,  the  underlying  performance 
information  should  be  credible,  sufficient,  and  used  by  all 
budget  decision  makers.  The  government  has  to  involve  the 
stakeholders  in  preparing  the  PBB  implementation, 
especially,  the  DPR  in  which  both  will  get  involved  in 
preparing  the  rules  and  regulations  so  that  this  effort 
would  not  be  a  hollow  paperwork  exercise. 

More  importantly,  to  emulate  the  success  of  GPRA,  the 
government  and  DPR  has  to  enact  a  new  law  for  the  blueprint 
or  detailed  content  of  PBB  to  be  applied  in  government 
departments  and  agencies  so  that  they  would  be  challenged 
and  strongly  encouraged  to  appropriately  take  the  PBB 
effort  as  a  priority.  The  role  of  the  legislature  (DPR)  in 
cooperating  with  government  is  very  critical.  Not  only 
because  DPR  has  to  get  involved  for  the  preparation  of  the 
law  as  mentioned  above  but  also  because  it  is  DPR  that 
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would  significantly  employ  the  PBB  in  the  process  of  budget 
approval  and  evaluation. 

The  decision  from  the  government  to  fully  implement  PBB 
for  the  FY2006  perhaps  should  be  intensely  evaluated. 
Currently,  the  biggest  challenge  to  implement  the  PBB  is 
lack  of  understanding  of  the  concept  of  strategic  planning 
and  performance  measures;  even  many  public  managers  do  not 
understand  performance  measures  such  as  the  difference 
between  output  and  outcome.  Selecting  several  pilot  projects 
in  each  department  or  agency  might  be  more  effective  so  that 
any  deviation  and  suggestion  for  improvement  can  be  quickly 
realized. 

The  opportunities  by  having  the  Law  No.  25/2004  on 
National  Development  Planning  System  have  to  be  exploited  to 
better  result  in  the  departments  and  agencies  strategic 
planning.  Instruction  for  the  preparation  has  to  be  well 
designed  and  clearly  enunciated  so  that  departments  and 
agencies  can  provide  strategic  framework  that  actually  can 
be  linked  to  day  to  day  activities  and  inspire  all  the 
elements  in  the  organization  to  shift  to  the  new  paradigm 
and  new  way  doing  of  business.  The  instruction  is  very 
important  because  it  is  not  possible  that  they  would  capture 
the  old  long  term  planning  frameworks  from  GBHN.  In  this 
matter,  the  Medium  Term  Expenditure  Framework  (MTEF)  aimed 
to  enhance  the  inter-correlation  between  planning  and 
budgeting  process  and  to  obtain  more  rational  and  strategic 
resources  allocation. 

In  selecting  the  performance  measures  that  would  be 
applied,  the  measures  have  to  be  adjusted  to  the  need  of  the 
country  as  a  developing  country.  Specifically,  although 
efficiency  and  effectiveness  have  been  dominantly  used  in 
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the  developed  countries  where  Indonesian  government  can 
obtain  the  benchmarking,  the  perspective  of  economy  must  be 
considered  as  the  first  priority.  As  a  developing  country 
which  has  limited  resources  to  be  allocated,  the  aspect  of 
economy  to  better  obtain  the  lowest  price  among  alternatives 
as  well  as  cost  reduction  on  current  expenditures  would 
provide  cost  saving  and  optimum  resource  allocation. 
Additionally,  due  to  the  lack  of  performance  standards  and 
benchmarking  available  for  the  public  sector,  the  method 
used  in  GPRA  by  having  annual  performance  plans  and  annual 
performance  report  to  be  compared  in  performance  evaluation 
stages  can  be  emulated. 

Especially  in  DOD/TNI,  the  economy  perspective  is  very 
critical  due  to  many  procurements  funded  by  Overseas  Export 
Credit  with  its  high  lobbying  cost,  and  short  grace  loan 
period,  and  high  interest  rates,  and  the  new  approach  from 
Indonesian' s  government  in  developing  national  military 
industries  to  create  and  develop  main  weaponry  system.  The 
more  cost  reduction  obtained  from  the  lower  price  on 
procurement,  the  more  optimum  budget  allocation  can  be 
obtained,  and  finally  the  more  weapons  can  be  procured  by 
using  the  same  amount  budget. 

The  current  system  cost  per  unit  measures  such  as  cost 
per  flying  hour,  cost  per  steaming  hour,  cost  per  tank  mile 
that  have  been  used  in  preparing  the  current  budget  might 
be  considerations  in  selecting  the  costing  method  for  the 
performance  based  budgeting  between  the  full  cost  method 
and  the  ABC  method.  Additional  work  to  be  done  is  to 
establish  a  blueprint  that  provides  manual  instruction  of 
the  ABC  and  as  basis  for  the  creating  a  performance  based 
costing . 
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Attention  also  must  be  focused  on  the  strategic 
planning  for  DOD  and  TNI  that  must  contain  the  same 
mission,  vision,  goals  and  objectives,  and  strategies  as 
the  inspirational  framework  in  doing  the  business  and 
developing  the  defense  sector  as  well  as  the  national 

military  industries.  It  is  time  for  both  sides  to  sit 
together,  get  rid  of  the  sectoral  ego,  and  think 

strategically  on  how  to  build  a  solid  institution  and  to 
give  priority  to  the  professionalism  as  the  guardian  of  the 
nation' s  freedom  and  safety  as  well  as  to  complete  the 

internal  reformation  in  the  defense  sector. 

B.  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

This  study  provides  an  overview  of  performance  based 
budgeting  and  a  basic  model  for  Indonesian  DOD/TNI.  The 
study' s  intent  is  to  promulgate  the  Indonesian  DOD/TNI  to 
better  prepare  the  further  requirements  of  the  performance 
based  budgeting  process  that  would  be  implemented  in  2006. 
Within  the  study  performance  based  budgeting  in  Indonesian 
defense  sector,  there  are  several  areas  to  conduct 

supporting  research  as  following: 

•  How  specifically  the  performance  based 
budgeting  would  assist  the  stakeholders 
decision  making  in  allocating  the  limited 
resources  ? 

•  How  performance  could  specifically  be  measured 
and  differed  in  terms  of  war  deployment  and 
peacetime  operation? 

•  Within  Indonesian  DOD/TNI  how  specifically  the 
performance  would  be  measured  as  well  as  how 
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the  performance  measurement  system  would  be 
appropriately  designed? 

•  How  the  Activity  Based  Costing  (ABC)  would  be 
used  as  a  costing  method  in  DOD/TNI? 

•  How  the  performance  based  budgeting  concept 
would  be  linked  to  the  business  activities 
within  Indonesian  DOD/TNI  such  as  in 
acquisition  (performance  based  acquisition)  and 
in  logistic  (performance  based  logistic) . 
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APPENDIX  A.  PROGRAM  ASSESSMENT  RATING  TOOL  TYPES  OF 

PROGRAMS 


(Source:  From  0MB  PART  Instruction  FY  2006) 
TYPES  OF  PROGRAMS 


Although  most  PART  questions  are  the  same,  the  PART  divides 
all  programs  into  seven  categories  for  the  purpose  of 
asking  additional  questions  unique  to  a  particular  type  of 
program.  These  categories  apply  to  both  discretionary  and 
mandatory  programs . 

1.  Direct  Federal  Programs  where  services  are  provided 
primarily  by  employees  of  the  Federal  government,  such  as 
the  National  Weather  Service  and  the  Visa  and  Consular 
Services . 

2.  Competitive  Grant  Programs  that  provide  funds  to  State, 
local  and  tribal  governments,  organizations,  individuals 
and  other  entities  through  a  competitive  process,  such  as 
Health  Centers. 

3.  Block/Formula  Grant  Programs  that  provide  funds  to  State, 
local  and  tribal  governments  and  other  entities  by 
formula  or  block  grant,  such  as  Weatherization  Assistance 
and  the  Ryan  White  program. 

4.  Regulatory  Based  Programs  that  accomplish  their  mission 
through  rulemaking  that  implements,  interprets  or 
prescribes  law  or  policy,  or  describes  procedure  or 
practice  requirements,  such  as  the  Food  Safety  and 
Inspection  Service. 

5.  Capital  Assets  and  Service 

Acquisition  Programs  that  achieve  their  goals  through 
development  and  acquisition  of  capital  assets  (such  as 
land,  structures,  equipment,  and  intellectual  property) 
or  the  purchase  of  services  (such  as  maintenance,  and 
information  technology)  ,  for  example.  Defense 
Shipbuilding  and  the  Bonneville  Power  Administration. 


131 


6.  Credit  Programs  that  provide  support  through  loans,  loan 
guarantees  and  direct  credit,  such  as  Export-Import 
Bank/Long  Term  Guarantees. 

7.  Research  and  Development 

(R&D)  Programs  that  focus  on  knowledge  creation  or  its 
application  to  the  creation  of  systems,  methods, 
materials,  or  technologies,  such  as  the  Department  of 
Energy/Solar  Energy  and  NASA/Mars  Exploration  programs. 


There  is  a  separate  PART  form  for  each  of  the  seven  types 
of  Federal  programs,  though  most  of  the  questions  are 
common  across  the  seven  forms.  The  vast  majority  of  Federal 
programs  fit  into  one  of  the  seven  categories  of  programs 
for  which  there  is  a  PART.  However,  some  programs  use  more 
than  one  mechanism  to  achieve  their  goals  (e.g.,  grants  and 
credit)  .  Even  in  these  cases,  using  one  PART  is  likely  to 
be  sufficient.  To  enable  this  for  R&D  programs,  which  can 
use  one  of  the  other  program  types  (e.g.,  competitive 
grants)  as  a  means  of  funding  R&D,  the  R&D  PART  has  been 
designed  to  enable  R&D  programs  that  fund  research  through 
grants,  contracts,  cooperative  agreements  or  other 
transactions  to  answer  questions  from  the  Competitive 
Grants  instrument.  Similarly,  R&D  programs  that  construct 
or  operate  equipment  or  facilities  will  answer  some 
questions  from  the  Capital  Assets  and  Service  Acquisition 
PART.  There  may  be  other  cases  in  which  drawing  questions 
from  two  different  PARTs  -  i.e.,  creation  of  a  "mixed"  form 
-  yields  a  more  informative  assessment.  In  those  instances, 
the  PART  type  that  most  closely  reflects  the  core  functions 
of  the  program  should  be  chosen  as  a  base,  and  then,  if 
necessary,  selected  questions  from  another  PART  can  be 
added.  The  0MB  examiner  should  consult  with  a  member  of  the 
0MB  Performance  Evaluation  Team,  if  considering  this 
approach . 

For  new  programs  for  which  it  is  impractical  to  expect 
results,  it  is  possible  to  complete  only  Sections  1  through 
III  of  the  PART.  However,  performance  measures,  targets  and 
related  information  should  still  be  provided  (and  scored) 
in  Section  IV  for  new  programs  where  practical. 

Question-specific  instructions  are  attached  to  help  explain 
the  purpose  of  each  question  and  general  standards  for 
evaluation.  The  individual  PART  worksheets  also  contain 
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this  guidance  as  well  as  instructions  on  the  technical 
aspects  of  using  the  worksheets.  These  instructions  will 
not  cover  every  case,  and  it  is  up  to  the  user  to  bring 
relevant  information  to  bear  in  answering  each  question 
that  will  contribute  to  the  program's  assessment. 
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APPENDIX  B.  PROGRAM  ASSESSMENT  RATING  TOOL 

QUESTIONS 


(Source:  0MB  PART  Instruction  FY  2006) 

I.  PROGRAM  PURPOSE  &  DESIGN 

This  section  examines  the  clarity  of  program  purpose  and 
soundness  of  program  design.  It  looks  at  factors  including 
those  the  program,  agency,  or  Administration  may  not 
directly  control  but  which  are  within  their  influence,  such 
as  legislation  and  market  factors.  Programs  should 
generally  be  designed  to  address  a  either  an  efficiency 
matter,  such  as  a  public  good  or  externality,  or  a 
distributional  objective,  such  as  assisting  low-income 
families  in  the  least  costly  or  most  efficient  manner.  A 
clear  understanding  of  program  purpose  is  essential  to 
setting  program  goals,  measures,  and  targets;  maintaining 
focus;  and  managing  the  program.  Potential  source  documents 
and  evidence  for  answering  questions  in  this  section 
include  authorizing  legislation,  agency  strategic  plans, 
performance  plans/performance  budgets,  and  other  agency 
reports . 

Options  for  answers  are  Yes,  No  or  Not  Applicable.  Design 
flaws  in  the  underlying  legislation  can  and  should  be 
considered  and  supported  by  evidence,  and  are  grounds  for  a 
No.  Not  Applicable  answers  are  likely  to  be  rare, 
particularly  for  items  1.1-1. 4,  as  these  questions  should 
apply  to  virtually  all  programs.  (For  R&D  programs,  most  of 
the  questions  in  this  section  help  address  program 
"relevance, "  one  of  the  three  fundamental  issues  of  the  R&D 
Investment  Criteria  (see  Appendix  A) .) 

1.1  Is  the  program  purpose  clear? 

Purpose  of  the  question:  to  determine  whether  the  program 
has  a  focused  and  well-defined  mission.  Determining  this 
purpose  is  critical  to  determination  of  useful 
performance  measures  and  targets. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  a 
clear  and  unambiguous  mission.  Considerations  can  include 
whether  the  program  purpose  can  be  stated  succinctly.  A 
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No  answer  would  be  appropriate  if  the  program  has 
multiple  conflicting  purposes. 

Evidence/Data :  Evidence  can  include  a  statement  of  the 
purpose  and  supporting  objectives  from  the  program's 
authorizing  legislation,  program  documentation  or  mission 
statement . 

1.2  Does  the  program  address  a  specific  and  existing 
problem,  interest,  or  need? 

Purpose  of  the  question:  to  determine  whether  the  program 
addresses  a  specific  problem,  interest,  or  need  that  can 
be  clearly  defined  and  that  currently  exists. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  the 
existence  of  a  relevant  and  clearly  defined  interest, 
problem  or  need  that  the  program  is  designed  to  address. 
A  Yes  answer  would  also  require  that  the  program  purpose 
is  still  relevant  to  current  conditions  (i.e.,  that  the 
problem  the  program  was  created  to  address  still  exists)  . 
Considerations  could  include,  for  example,  whether  the 
program  addresses  a  specific  market  failure.  A  No  should 
be  given  if  there  is  no  clear  need  for  the  program. 

Programs  may  receive  a  Yes  to  question  1 . 1  and  a  No  on 
question  1.2  and  vice  versa. 

For  example.  Housing  and  Urban  Development's  HOPE  VI 
program  had  a  clear  purpose  with  a  goal  of  demolishing 
100,000  of  the  most  severely  distressed  public  housing 
units.  However,  the  program  has  surpassed  this  goal  and 
addressed  the  need  for  which  the  program  was  originally 
created.  Therefore,  the  program  received  a  Yes  in  1.1  and 
a  No  in  1.2.  In  addition,  the  Department  of  Education 
Vocational  Education  program  had  an  unclear  mission 
caused  by  multiple  and  overlapping  objectives.  It  was 
able  to  document,  however,  that  a  significant  number  of 
students  are  graduating  from  high  school  and  community 
college  without  the  necessary  academic  and  technical 
skills.  Therefore,  the  program  received  a  No  for  1.1  and 
a  Yes  for  1.2. 

Evidence/Data:  Evidence  should  describe  the  problem, 
interest  or  need  that  the  program  is  designed  to  address 
and  include  relevant  documentation.  An  example  could  be 
the  number  and  income  levels  of  uninsured  individuals  for 
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a  program  that  provides  care  to  those  without  health 
insurance . 

1.3  Is  the  program  designed  so  that  it  is  not  redundant  or 
duplicative  of  any  other  Federal,  State,  local  or  private 
effort? 

Purpose  of  the  question:  to  determine  whether  the  program 
is  designed  to  fill  a  unique  role  or  whether  it  instead 
unnecessarily  duplicates  or  even  competes  with  other 
Federal  or  non-federal  programs. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  that 
the  program  does  not  excessively  overlap  with  other 
Federal  or  non-federal  efforts,  including  the  efforts  of 
State  and  local  governments  or  the  private  and  non-profit 
sectors.  A  consideration  can  include  whether  the  program 
serves  a  population  not  served  by  other  programs. 

A  No  answer  should  be  given  when  there  is  more  than  one 
program  that  addresses  the  same  problem,  interest,  or 
need,  regardless  of  the  size  or  history  of  the  respective 
programs.  For  programs  that  partially  overlap  with 
others,  a  No  should  be  given  when  major  aspects  of  the 
program,  such  as  its  purpose,  targeted  beneficiaries,  or 
mechanisms,  are  duplicative.  If  there  are  two  programs 
that  significantly  overlap  and  one  is  large  and  another 
is  small,  both  programs  should  receive  a  No  for  this 
question . 

Similar  programs  might  be  justified  in  receiving  a  Yes  if 
a  strong  case  can  be  made  that  fixed  costs  are  low  and 
competition  is  beneficial  (e.g.,  perhaps  multiple 
laboratories)  or  if  more  than  one  service  delivery 
mechanism  is  appropriate  (e.g.,  block  grants  for  base 
activities  and  competitive  grants  for  demonstration 
projects) .  Also,  the  standard  of  evidence  to  receive  a 
Yes  should  be  higher  where  Federal  programs  overlap  with 
each  other  than  where  a  Federal  program  overlaps  with 
private,  local,  or  State  programs.  For  example,  two 
Federal  programs  to  address  training  would  face  a  high 
standard  to  receive  a  Yes  to  this  question;  in  contrast, 
a  Yes  could  be  provided  to  a  Federal  program  to  address 
indigent  medical  care  across  the  nation,  even  though 
there  are  many  local  and  private  programs  that  also 
address  indigent  medical  care.  The  key  would  be  whether 
the  gaps  in  the  non-Federal  provision  are  large  enough  to 


137 


warrant  a  Federal  program  and  whether  the  Federal  program 
is  well  designed  to  mesh  with  non-Federal  efforts  and 
responsibilities . 

For  credit  programs,  a  Yes  answer  would  require  evidence 
of  the  market  failure/absence  or  unwillingness  of 
private-sector  participation  and  an  overview  of  the 
market,  including  all  international.  Federal,  local,  and 
private-sector  participants. 

For  R&D  programs,  some  degree  of  duplication  is 
permissible,  if  it  is  well  justified  and  coordinated.  A 
Yes  answer  would  require  justification  that  the  program 
provides  value  beyond  that  of  any  similar  efforts  at  the 
agency,  efforts  at  other  agencies,  or  efforts  funded  by 
State  and  local  government,  private  and  non-profit 
sectors,  or  other  counties.  Justification  first  requires 
due  diligence  in  identifying  similar  past  or  present 
efforts . 

Evidence/Data :  Evidence  should  identify  duplicative 
programs  and  their  total  expenditures  and/or  a 
description  of  efforts  supported  by  those  programs  that 
address  a  similar  problem  in  a  similar  way  as  the  program 
being  evaluated. 

1.4.  Is  the  program  design  free  of  major  flaws  that  would 

limit  the  program' s  effectiveness  or  efficiency? 

Purpose  of  the  question:  to  determine  whether  there  are 
major  design  flaws  in  the  program  that  limit  its 
efficiency . 

Elements  of  a  Yes  answer:  The  program  should  be  free  from 
major  design  flaws  that  prevent  it  from  meeting  its 
defined  objectives  and  performance  goals.  To  receive  a 
Yes,  there  should  be  no  strong  evidence  that  another 
approach  or  mechanism  would  be  more  efficient  or 
effective  to  achieve  the  intended  purpose.  A 
consideration  could  be  whether  the  government  would  get 
the  same  or  better  outcome  by  expending  fewer  total 
resources  through  a  different  mechanism.  For  example, 
there  may  be  evidence  that  a  regulatory  program  to  ensure 
public  safety  would  be  more  effective  than  a  grant 
program.  Analysis  should  consider  whether  the  program 
structure  continues  to  make  sense  given  changing 
conditions  in  the  field  (e.g.,  changing  threat  levels  or 
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social  conditions)  .  Other  considerations  could  include 
whether  the  program  extends  its  impact  by  leveraging 
funds  and  contributions  from  other  parties. 

For  credit  programs,  an  additional  consideration  can 
include  the  extent  to  which  a  large  number  of  borrowers 
would  otherwise  not  have  access  to  financial  resources. 
Also  consider  whether  the  program  costs  are  adequate,  but 
not  excessive,  to  achieve  the  policy  goals.  For  example, 
a  Yes  answer  could  mean  that  the  program  effectively  uses 
market  mechanisms  to  reduce  government  risk  and  thus 
minimize  program  costs. 

Regulatory  programs  should  receive  a  No  if  the  statute 
underlying  the  regulations  is  not  designed  to  maximize 
net  benefits. 

For  capital  assets  and  service  acquisition  programs,  a 
Yes  answer  requires  that,  in  addition  to  the  general 
criteria,  the  program  is  supported  by  an  adequate  capital 
asset  management  infrastructure  that  is  consistent  with 
the  principles  and  techniques  of  effective  capital 
programming,  and  has  clear  lines  of  authority, 
responsibility,  and  accountability  for  managing  capital 
assets . 

Evidence/Data :  Evidence  demonstrating  efficient  design 
can  include  cost  effectiveness  studies  comparing 
alternative  mechanisms  (e.g.,  regulations  or  grants)  with 
the  current  design  (say,  direct  federal  provision) . 
Evidence  on  the  relative  benefits  and  costs  of  the 
activity  are  also  useful. 

Evidence  for  determining  whether  the  threshold  for 
capital  programming  has  been  met  should  include  the 
documented  program-relevant  agency  or  bureau  capital 
programming  policies,  directives,  instructions,  manuals, 
and  assignment  of  authorities  and  responsibilities  to 
agency  personnel  and  organizational  units. 

1.5  Is  the  program  design  effectively  targeted,  so  that 
resources  will  reach  intended  beneficiaries  and/or 
otherwise  address  the  program' s  purpose  directly? 

Purpose  of  the  question:  to  determine  whether  the  program 
is  designed  so  that  program  resources  will  reach  the 
intended  beneficiaries  efficiently  and  to  avoid 
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unintended  subsidies.  "Beneficiaries"  refers  to  those  who 
benefit  from  the  favorable  outcome  of  the  program. 
"Reach"  refers  to  the  distribution  of  benefits. 

Unlike  Question  1.4,  which  addresses  examination  of 
alternatives  to  achieve  a  program's  goals,  this  question 
asks  whether  program  resources  under  the  chosen 
alternative  are  oriented  toward  the  effective  achievement 
of  the  program's  purpose. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  that 
a  program  demonstrates  that  the  right  beneficiaries  are 
being  targeted,  activities  that  would  have  occurred 
without  the  program  are  not  subsidized  (or  receive  only 
warranted  levels  of  subsidies)  ,  and  program  funds  are 
targeted  effectively  to  meet  program  purposes. 

Acceleration  of  activities  due  to  Federal  funding  can  be 
grounds  for  a  Yes,  but  there  should  be  evidence  that  the 
acceleration  warrants  the  subsidy  or  application  of 
funding.  Acceleration  of  an  activity  that  increases 
profits  for  a  business  --  that  the  firm  would  or  could 
have  undertaken  eventually  without  the  subsidy  or 
application  of  funding  --  would  not  generally  qualify  for 
a  Yes,  unless  there  are  significant  external  (i.e., 
social)  benefits  from  the  activity. 

In  the  case  of  block-grant  and  credit  programs,  the 
assessment  should  also  consider  how  well  funds  are 
targeted  to  meet  the  program  purpose  and  whether  funds 
are  protected  against  supplantation  or  substitution. 

For  R&D  programs,  a  Yes  answer  would  require 
identification  of  relevance  to  specific  national  needs, 
agency  missions,  fields  of  science  or  technology,  or 
other  "customer"  needs.  A  customer  may  be  another  program 
at  the  same  or  another  agency,  an  interagency  initiative 

or  partnership,  a  firm_^^ _ or  an  organization  from  another 

sector  or  country.  For  these  programs,  the  question 
refers  to  awardees  of  contracts,  cooperative  agreements 
or  other  transactions,  as  well  as  grants. 

Evidence/Data:  Evidence  should  show  that  the  program  is 
designed  to  1)  reach  the  highest  practicable  percentage 
of  target  beneficiaries,  and  2)  have  the  smallest 
practicable  share  of  funds  going  to  unintended 
beneficiaries.  Regarding  item  1,  a  small  program  may  only 
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be  able  to  reach  a  small  number  of  beneficiaries,  but  it 
should  be  well  targeted  on  some  merit  basis.  On  item  2, 
programs  not  designed  to  avoid  unwarranted  shares  of 
funding  going  to  beneficiaries  who  do  not  need  or  merit 
the  funding  should  receive  a  No  answer.  Programs  that  are 
designed  in  a  way  that  is  likely  to  result  in  significant 
levels  of  erroneous  payments  should  receive  a  No. 


II.  STRATEGIC  PLANNING 

This  section  focuses  on  program  planning,  priority  setting, 
and  resource  allocation.  Key  elements  include  an  assessment 
of  whether  the  program  has  a  limited  number  of  performance 
measures  with  ambitious  --  yet  achievable  --  targets,  to 
ensure  planning,  management,  and  budgeting  are  strategic 
and  focused.  Potential  source  documents  and  evidence  for 
answering  questions  include  strategic  planning  documents, 
agency  performance  plans/performance  budgets  and  reports, 
reports  and  submissions  from  program  partners,  evaluation 
plans,  and  other  program  documents. 

Options  for  answers  are  Yes,  No  or  Not  Applicable.  While  it 
is  recognized  that  some  programs  may  have  greater 
difficulty  than  others  in  developing  quantitative 
performance  goals,  programs  must  have  meaningful  and 
appropriate  methods  for  demonstrating  results.  0MB  and 
agencies  should  work  together  to  develop  approaches  for 
programs  where  it  is  difficult  to  develop  quantitative 
measures,  and  where  qualitative,  expert-review,  or  other 
measures  are  more  appropriate.  For  R&D  programs,  most 
questions  in  this  section  help  address  the  prospective 
aspects  of  the  R&D  Investment  Criteria  (see  Appendix  A) . 

2.1  Does  the  program  have  a  limited  number  of  specific 
long-term  performance  measures  that  focus  on  outcomes  and 
meaningfully  reflect  the  purpose  of  the  program? 

Purpose  of  the  question:  to  determine  if  the  program  has 
long-term  performance  measures  to  guide  program 
management  and  budgeting  and  promote  results  and 
accountability.  This  question  seeks  to  assess  whether  the 
program  measures  are  salient,  meaningful,  and  capture  the 
most  important  aspects  of  program  purpose  and  appropriate 
strategic  goals. 
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Elements  of  a  Yes  answer:  A  Yes  requires  two  or  three 
specific,  easily  understood  outcome  measures  that 
directly  and  meaningfully  support  the  program's  purpose. 
"Long-term"  means  a  long  period  relative  to  the  nature  of 
the  program,  likely  5-10  years,  and  consistent  with  time 
periods  for  strategic  goals  used  in  the  agency' s  GPRA 
strategic  plan. 

The  performance  measures  should  focus  mainly  on  outcomes, 
although  in  some  cases  output  measures  are  permissible. 
The  measures  may  be  those  developed  to  comply  with  GPRA, 
so  long  as  they  meet  the  "Performance  Measures"  section 
of  this  document.  Otherwise,  0MB  and  agencies  should 
revise  the  measures  to  fully  meet  GPRA  and  PART 
standards.  A  Yes  answer  can  also  be  given  if  0MB  and  the 
agency  have  reached  agreement  on  a  limited  number  of 
long-term  measures  that  will  be  added  to  the  2006  GPRA 
strategic  plan  or  performance  plan/performance  budget. 
Significant  changes  to  the  GPRA  strategic  plan  may 
require  stakeholder  consultation. 

Output  measures  only  meet  the  standards  of  a  Yes  answer 
if  the  program  can  produce  sound  justification  for  not 
adopting  outcome  measures.  For  example,  a  program  that  is 
exclusively  focused  on  processing  applications  and  is 
unable  to  adequately  define  a  quantifiable  outcome 
measure  may  use  measures  that  focus  on  increases  in 
accuracy  and/or  timeliness  of  service  delivery.  Whenever 
output  measures  are  proposed,  the  program  must  clearly 
show  how  such  measures  reflect  progress  toward  desired 
outcomes . 

An  example  of  an  unacceptable  long-term  measure  is  a 
housing  program  that  is  proposing  using  a  measure  of 
"number  of  housing  units  constructed."  In  this  case,  such 
an  output  is  unacceptable  as  the  program  is  expected  to 
articulate  and  measure  progress  toward  achieving  outcomes 
(e.g.,  increases  in  homeownership  rates,  increases  in 
housing  equity  in  low-income  communities) . 

For  more  detailed  discussion  on  when  output  measures  may 
be  used  as  proxy  measures  for  outcomes,  please  see 
"Selecting  Performance  Measures"  section  of  the  PART 
guidance  or  visit  0MB' s  website  at 
http : / / WWW . omb . gov/ part/  . 


142 


A  No  must  be  given  for  long-term  measures  that  do  not 
directly  and  meaningfully  relate  to  the  program' s  purpose 
or  are  unnecessarily  focused  on  outputs  and  lack  adequate 
justification.  A  program  must  not  receive  a  No  for  having 
too  many  measures,  if  it  has  identified  a  few  high- 
priority  ones  that  represent  important  aspects  of  the 
program. 

Performance  measures  should  be  listed  in  the  Measures  tab 
of  the  PART  worksheet.  Only  measures  that  meet  the 
standards  for  a  Yes  should  be  entered  on  the  worksheet. 

For  block  grant  programs  that  support  a  wide  range  of 
purposes  and  allow  grantees  to  set  their  own  program 
priorities,  measures  that  address  the  extent  to  which 
grantees  meet  their  own  goals  or  effectively  target 
populations  are  options  if  no  better  measures  are 
possible . 

For  R&D  programs,  0MB  will  work  with  agencies  to  assess 
existing  or  develop  appropriate  measures.  Some  R&D 
programs,  especially  in  basic  research,  may  not  be 
required  to  provide  long-term  efficiency  measures,  due  to 
the  uncertainty  of  outcomes  and  the  years  it  takes  to 
achieve  and  recognize  them. 

Evidence/Data :  Evidence  will  include  the  long-term 
measures  established  for  the  program  either  in  the 
existing  agency  GPRA  documents  or  other  program  documents 
or  as  agreed  to  by  0MB  and  to  be  included  in  the  2006 
GPRA  documents.  In  the  case  of  new  measures,  if  targets 
and  baselines  are  not  defined,  a  plan  for  their 
development  (i.e.,  timeline,  methods  for  data  collection, 
responsible  office  and/or  staff)  must  be  agreed  to  by  the 
agency  and  0MB. 

2.2  Does  the  program  have  ambitious  targets  and  timeframes 

for  its  long-term  measures? 

Purpose  of  the  question:  to  determine  if  the  program  has 
challenging  but  realistic  quantifiable  targets  and 
timeframes  for  the  long-term  measures. 

Elements  of  a  Yes  answer:  A  Yes  answer  requires  that 
specific  quantified  targets  have  been  developed  for 
measures  evaluated  in  Question  2.1.  (Where  targets  are 
not  "quantitative,"  they  still  must  be  verifiable,  e.g.. 
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through  a  clean  audit  or  outstanding  ratings  by  an  expert 
panel) .  Baselines  from  which  to  measure  targets  are 
encouraged  for  outcome  measures  and  required  for  output 
measures.  Targets  and  timeframes  must  be  ambitious,  that 
is,  they  must  be  set  at  a  level  that  promotes  continued 
improvement  within  achievable  efficiencies.  Where 
relevant,  a  Yes  also  requires  that  a  program  has  defined 
an  appropriate  end  target. 

A  No  is  appropriate  if  quantified  targets  or  timeframes 
are  not  included  for  key  measures  or  if  the  targets  or 
timeframes  are  not  ambitious  or  challenging. 

If  the  program  received  a  No  in  Question  2.1,  the  program 
must  get  a  No  for  this  question. 

Targets  must  be  listed  in  the  Measures  tab  of  the  PART 
worksheet . 

For  R&D  programs,  a  Yes  answer  would  require  that  the 
program  provides  multi-year  R&D  objectives.  Where 
applicable,  programs  must  provide  schedules  with  annual 
milestones,  highlighting  any  changes  from  previous 
schedules.  Program  proposals  must  define  what  outcomes 
would  represent  a  minimally  effective  program  and  a 
successful  program. 

Evidence/Data:  Evidence  will  include  targets  in  existing 
agency  GPRA  documents  or  other  program  documents  or  as 
agreed  to  by  0MB  and  to  be  included  in  the  2006  GPRA 
documents . 

2.3  Does  the  program  have  a  limited  number  of  specific 
annual  performance  measures  that  can  demonstrate  progress 
toward  achieving  the  program's  long-term  goals? 

Purpose  of  the  question:  to  determine  whether  a  limited 
number  of  annual  performance  measures  have  been 
identified  that  directly  support  the  long-term  goals 
evaluated  in  Questions  2.1  and  2.2.  The  measures  should 
be  logically  linked  to  the  long-term  goals  in  a  manner 
that  enables  them  to  demonstrate  progress  toward 
achieving  those  long-term  goals. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require 
annual  performance  measures  that  are  discrete, 
quantifiable,  and  measurable.  Most  importantly,  these 
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annual  measures  should  measure  the  program's  progress 
toward  reaching  the  long-term  goals  evaluated  in 
Questions  2.1  and  2.2.  The  annual  performance  measures 
may  focus  on  outputs  and  may  or  may  not  be  those 
developed  by  the  agency  to  comply  with  GPRA.  A  Yes  answer 
can  also  be  given,  if  0MB  and  the  agency  have  reached 
agreement  on  a  limited  number  of  annual  measures  that 
will  be  included  in  the  2006  GPRA  performance 
plan/performance  budget. 

To  receive  a  Yes  answer,  programs  must  have  at  least  one 
efficiency  measure  or  be  developing  one.  If  a  measure  is 
under  development,  a  schedule  for  adoption  (i.e., 
timeline,  methods  for  data  collection,  responsible  office 
and/or  staff)  must  be  agreed  to  by  the  agency  and  0MB. 

For  more  detailed  discussion  on  defining  acceptable 
efficiency  measures  please  see  "Selecting  Performance 
Measure"  section  of  the  PART  guidance  or  visit  0MB' s 
website  at  http://www.omb.gov/part/  . 

Programs  that  are  unable  to  provide  an  efficiency  answer 
may  only  receive  a  Yes  to  this  question  if  they  are  able 
to  adequately  explain  why  adoption  of  such  a  measure  is 
not  feasible  (e.g.,  some  basic  R&D  programs). 

If  the  program  received  a  No  in  Question  2.1,  an 
explanation  of  how  annual  performance  goals  contribute  to 
desired  long-term  outcomes  and  purpose  of  the  program 
must  be  provided  to  receive  a  Yes  for  this  question. 
Performance  measures  should  be  listed  in  the  Measures  tab 
of  the  PART  worksheet.  Only  measures  that  meet  the 
standards  for  a  Yes  should  be  entered  on  the  worksheet. 

For  block  grant  programs  that  support  a  wide  range  of 
purposes  and  allow  grantees  to  set  their  own  program 
priorities,  measures  that  address  the  extent  to  which 
grantees  meet  their  own  goals  or  effectively  target 
populations  are  options  if  no  other  measures  are 
possible . 

For  R&D  programs,  a  Yes  answer  would  require  that  the 
program  has  annual  performance  measures  to  track  how  the 
program  could  improve  scientific  understanding  and  its 
application.  For  R&D  programs  that  have  multi-year 
schedules,  the  annual  measure  should  tie  into  the  longer 
term  milestones,  as  appropriate.  Some  basic  research 


145 


programs  may  not  be  able  to  define  meaningful  annual 
outcome  or  efficiency  measures.  In  such  cases,  these 
programs  may  use  process-related  indicators  (see  Question 
3.4),  if  the  program  can  explain  how  those  processes  are 
directed  toward  the  intended  long-term  goals.  0MB  will 
work  with  agencies  to  address  appropriate  measures. 

For  capital  assets  and  service  acquisition  programs,  a 
Yes  answer  requires  that,  in  addition  to  the  general 
criteria,  annual  performance  measures  include  those  that 
are  sufficient  to  track  achievement  of  the  cost, 
schedule,  and  performance  goals  of  asset  acquisitions  as 
they  relate  to  the  overall  program. 

Evidence/Data:  Evidence  will  include  the  annual  measures 
established  for  the  program  in  the  agency  GPRA 
performance  plan/performance  budget  or  other  program 
documents,  or  they  may  be  new  measures  as  agreed  to  by 
0MB  and  which  will  be  included  in  the  2006  GPRA 
performance  plan/performance  budget.  Evidence  for  capital 
asset  acquisition  programs  includes  agency  acquisition 
and  project  management  working  documents,  contract 
performance  measures  and  metrics,  and  business  cases  (0MB 
Circular  A-11  Exhibit  300s)  . 

2.4.  Does  the  program  have  baselines  and  ambitious  targets 

for  its  annual  measures? 

Purpose  of  the  question:  to  determine  if  the  program  has 
baselines  and  challenging  but  realistic  quantified 
targets  for  the  annual  measures. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  generally 
require  that  baselines  have  been  established  for  most  of 
the  annual  measures  evaluated  in  Question  2.3.  A  Yes 
answer  also  requires  that  specific  quantified  annual 
targets  have  been  developed  for  most  of  the  program' s 
annual  measures  evaluated  in  Question  2.3.  These  targets 
provide  a  specific  value  with  which  performance  can  be 
compared.  These  targets  must  be  ambitious,  that  is  they 
must  be  set  at  levels  that  ensure  continued  improvement 
and  realization  of  efficiencies.  They  also  should  be 
within  reason  for  the  program  to  achieve. 

A  No  answer  would  be  appropriate  if  quantified  targets  or 
timeframes  are  not  included  for  most  measures  or  if  the 
targets  are  not  ambitious  or  challenging. 
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If  the  program  received  a  No  in  Question  2.3,  the  program 
must  get  a  No  for  this  question.  Targets  should  be  listed 
in  the  Measures  tab  of  the  PART  worksheet. 

Evidence/Data:  Evidence  will  include  targets  in  the 
agency  GPRA  performance  plan/performance  budget  or  other 
program  documents  or  as  agreed  to  by  0MB  and  will  be 
included  in  the  2006  GPRA  performance  plan/performance 
budget . 

2.5.  Do  all  partners  (including  grantees,  sub-grantees, 
contractors,  cost-sharing  partners,  and  other  government 
partners)  commit  to  and  work  toward  the  annual  and/or  long¬ 
term  goals  of  the  program? 

Purpose  of  the  question:  to  determine  whether  program 
efforts  carried  out  by  program  partners  also  support  the 
annual  and  long-term  performance  goals  of  the  program. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  that 
program  managers  strive  to  ensure  that  partners  support 
the  overall  goals  of  the  program  and  measure  and  report 
on  their  performance  as  it  relates  to  accomplishing  those 
goals.  Eor  example,  a  program  that  requires  all  grant 
agreements  and  contracts  to  include  performance  measures 
that  will  help  the  program  achieve  its  goals  and  monitor 
those  measures  would  receive  a  Yes. 

If,  however,  a  program  does  not  through  a  performance 
requirement  or  some  other  means  get  program  grantees  to 
link  their  activities  to  the  program's  goals,  a  No  would 
be  appropriate.  The  most  obvious  example  of  a  partner  is 
an  entity  receiving  program  funding.  While  a  program 
cannot  always  control  the  activities  of  its  partners,  it 
can  exert  influence  through  a  number  of  various 
mechanisms . 

If  the  program  received  a  No  for  both  Questions  2.1  and 
2.3,  the  program  must  receive  a  No  for  this  question. 

In  the  case  of  regulatory  programs,  all  regulated 
entities  are  not  necessarily  defined  as  program  partners. 

For  programs  funding  grants  for  basic  research,  a  Yes 
answer  can  be  achieved  if  the  program  solicitation 
explicitly  includes  the  program  goals,  and  grant 
applications  and  progress  reports  provide  sufficient 
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means  for  the  program  manager  to  assess  performance  and 
continuing  relevance. 

Evidence/Data:  Evidence  can  include  contracts  and  other 
documents  that  tie  contractor  performance  to  program 
goals,  as  well  as  other  procedures  the  program  uses  to 
get  partners  to  commit  to,  measure,  and  report  on 
performance  related  to  the  program's  goals. 

2.6.  Are  independent  evaluations  of  sufficient  scope  and 
quality  conducted  on  a  regular  basis  or  as  needed  to 
support  program  improvements  and  evaluate  effectiveness  and 
relevance  to  the  problem,  interest,  or  need? 

Purpose  of  the  question:  to  ensure  that  the  program  (or 
agency)  conducts  non-biased  evaluations  on  a  regular  or 
as-needed  basis  to  fill  gaps  in  performance  information. 
These  evaluations  should  be  of  sufficient  scope  to 
improve  planning  with  respect  to  the  effectiveness  of  the 
program.  (Eor  R&D  programs,  this  question  is  central  to 
prospective  planning  to  address  all  of  the  R&D  investment 
criteria  (see  Appendix  A) .) 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require 
regularly  scheduled  objective,  high  quality,  independent 
evaluations  that  examine  how  well  the  program  is 
accomplishing  its  mission  and  meeting  its  long-term 
goals.  A  Yes  answer  also  would  require  that  program 
evaluations  address  the  following  elements  of  quality, 
scope,  and  independence. 

Not  Applicable  is  not  an  option  for  this  question;  given 
the  flexibility  in  determining  what  constitutes  an 
evaluation,  all  programs  should  undergo  an  evaluation 
that  meets  the  following  elements  of  quality,  scope,  and 
independence.  Quality.  Evaluations  should  be  sufficiently 
rigorous  to  provide  information  on  the  effectiveness  of 
the  program  and,  for  programs  that  support  or  employ  a 
range  of  services  and  approaches,  information  on  the 
effectiveness  of  the  various  services  and  approaches.  To 
receive  a  Yes,  agencies  should  provide  evidence  that  they 
have  chosen  and  applied  evaluation  methods  that  provide 
the  most  rigorous  evidence  of  a  program's  effectiveness 
that  is  appropriate  and  feasible.  A  program  may  receive  a 
Yes  for  this  question  if  the  agency  and  0MB  determine 
that  existing  evaluations  do  not  provide  the  most 
rigorous  evidence  possible,  but  that  the  program  is  in 
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the  process  of  developing  new  evaluation  approaches  that 
will  provide  the  most  rigorous  evidence  possible  by  a 
specified  future  date. 

The  purpose  of  the  question  is  to  find  out  whether 
agencies  have  evaluations  that  assess  the  effectiveness 
of  the  program.  The  most  significant  aspect  of  program 
effectiveness  is  impact— the  outcome  of  the  program,  which 
otherwise  would  not  have  occurred  without  the  program 
intervention.  Where  it  is  feasible  to  measure  the  impact 
of  the  program,  randomized  controlled  trials  are 
generally  the  highest  quality,  unbiased  evaluation  to 
demonstrate  the  actual  impact  of  the  program.  However, 
these  studies  are  not  suitable  or  feasible  for  every 
program,  and  a  variety  of  evaluation  methods  may  need  to 
be  considered  because  Federal  programs  vary  so 
dramatically.  Other  types  of  evaluations,  including  well- 
designed  quasi-experimental  studies,  may  provide  useful 
information  about  the  impact  of  a  program  (but  should  be 
scrutinized  given  the  increased  possibility  of  an 
erroneous  conclusion)  and/or  can  help  address  how  or  why 
a  program  is  effective  (or  ineffective)  (i.e.,  meeting 
performance  targets,  achieving  efficiency,  fulfilling 
stated  purpose) . 

Overall,  evaluations  must  be  appropriate  to  the  type  of 
program.  Agencies  and  0MB  should  consult  evaluation 
experts,  either  in-house  and/or  external,  as  appropriate, 
when  deciding  what  type  of  evaluation  will  provide  the 
most  rigorous  evidence  appropriate  and  feasible.  The 
following  are  several  links  to  references  on  program 
evaluation  located  on  the  internet  that  are  not  intended 
to  be  exhaustive,  but  which  may  be  helpful: 

Program  Evaluation  Methods:  Measurement  and  Attribution 
of  Program  Results;  Treasury  Board  of  Canada, 
Secretariat;  1998.  (a  book  available  online) 
http : / /www. tbs-sct . gc . ca/ eval/pubs/meth/pem-mep_e .pdf 

Understanding  Impact  Evaluation;  The  World  Bank  Group, 
(a  web  site) 
http : / / WWW . worldbank . org/poverty/ impact/ index . htm 

"Program  Evaluation:  An  Evaluation  Culture  and 
Collaborative  Partnerships  Build  Agency  Capacity;"  GAO- 
03-454;  U.S.  General  Accounting  Office;  May  2003. 
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http : / / WWW . gao . gov/ docdblite/ summary . php?recf lag=&accno=A 
0  67  97&rptno=GAO-03-4  54 

"Performance  Measurement  and  Evaluation:  Definitions 
and  Relationships;"  GAO/GGD-98-2 6 ;  U.S.  General 
Accounting  Office;  April  1998. 
http : / / WWW . gao . gov/ docdblite/ summary . php?recf lag=&accno=l 
602  04&rptno=GGD-98-2  6 

"Designing  Evaluations;"  GAO/PEMD-10 . 1 . 4 ;  U.S.  General 
Accounting  Office;  May  1991. 
http://161.203.16.4/t2pbat7/144040.pdf 

Randomized  Controlled  Trials:  A  User's  Guide;  Jadad, 
Alejandro  A.;  BMJ  Books;  1998.  (a  book  available  online) 
http : / / WWW . bmjpg . com/ ret/ contents . html 

Research  Methods  Knowledge  Base;  Trochim,  William  M.  ; 
Cornell  University.  (a  web  site) 
http : // trochim. human . Cornell . edu/kb/ 

"Identifying  and  Implementing  Educational  Practices 
Supported  By  Rigorous  Evidence:  A  User  Eriendly  Guide;" 
U.S.  Department  of  Education;  December  2003. 
http : / / WWW . ed . gov/ rschstat/ research/pubs/ rigorousevid/ rig 
orousevid . pdf 

Scope.  This  question  looks  directly  at  whether  there  are 
evaluations  on  the  program' s  achievement  of  performance 
targets,  and  that  these  evaluations  examine  the 
underlying  cause  and  effect  relationship  between  the 
program  and  the  target.  In  cases  where  a  comprehensive 
evaluation  is  unnecessary  based  on  the  known 
effectiveness  of  an  intervention  and  performance  data  on 
the  program,  evaluations  that  fill  gaps  in  performance 
information  can  meet  the  elements  of  a  Yes  answer.  A 
program's  effectiveness,  including  impact,  also  may  be 
considered.  A  program  may  receive  a  Yes  for  this  question 
if  the  agency  and  examiner  determine  that  existing 
evaluations  are  inadequate,  but  the  program  is  in  the 
process  of  developing  an  appropriate  evaluation  to  be 
completed  by  a  specified  future  date. 

Evaluations  also  should  include  recommendations  on  how  to 
improve  the  program's  performance.  To  ensure  the  program 
continues  to  meet  its  performance  targets,  an  evaluation 
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should  be  scheduled  on  a  periodic  basis,  such  as  every 
two  to  five  years,  or  whatever  time  schedule  is 
reasonable  based  on  the  specific  program  and  its  mission 
and  goals. 

R&D  programs  also  should  undergo  independent  reviews  of 
relevance  to  their  agencies,  fields  of  science  or 
technology,  or  customers,  (e.g.,  of  process)  in  addition 
to  assessing  questions  of  performance.  These  reviews 
should  conclude  with  reports  documenting  the  findings  and 
recommendations.  A  "customer"  may  be  another  program  at 
the  same  or  another  agency,  an  interagency  initiative  or 
partnership,  or  a  firm,  an  organization  from  another 
sector  or  country,  or  the  general  public.  Industry¬ 
relevant  programs  may  use  industry  cost-sharing  of 
associated  projects  as  an  indicator  of  market-relevance, 
and  they  should  incorporate  industry  in  planning  and 
prioritization.  Reviews  should  be  rigorous  and  methodical 
and  be  a  critique  of  the  program's  methods,  results  and 
findings  by  others  in  the  field  with  requisite  training, 
expertise,  and  independence. 

Independence.  To  be  independent,  non-biased  parties  with 
no  conflict  of  interest  would  conduct  the  evaluation. 
Evaluations  conducted  by  the  program  itself  should 
generally  not  be  considered  "independent;"  however,  if 
the  agency  or  program  has  contracted  out  the  evaluation 
to  a  third  party  this  may  qualify  as  being  sufficiently 
independent.  Evaluations  conducted  by  an  agency's 
Inspector  General  or  program-evaluation  office  might  also 
be  considered  "independent."  0MB  examiners  and  agency 
staff  will  determine  if  a  specific  evaluation  can  be 
considered  "independent"  for  this  question. 

If  a  program  has  had  previous  evaluations  that  meet  the 
elements  of  independence  and  scope,  but  that  were  not 
sufficiently  rigorous,  and  the  program  is  developing  or 
about  to  conduct  a  new  program  evaluation  using  the  most 
rigorous  method  that  is  feasible  and  appropriate,  then 
the  program  would  receive  a  Yes. 

In  the  absence  of  ANY  independent  evaluations,  a  program 
would  receive  a  No,  unless  it  is  planning  to  carry  out  a 
rigorous  program  evaluation  in  the  near  future.  A  No 
answer  would  also  be  appropriate  for  a  program  that  has 
insufficient  independent  evaluation  data  or  has 
evaluations  that  address  process  and  not  performance. 
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Evidence/Data:  Evidence  should  include  a  program 

evaluation  plan  or  schedule  of  program  evaluations  and 
program  documentation  describing  the  type  of  evaluation, 
including  scope  and  quality,  and  the  criteria  for 
selecting  an  independent  evaluator. 

2.7.  Are  Budget  requests  explicitly  tied  to  accomplishment 
of  the  annual  and  long-term  performance  goals,  and  are  the 
resource  needs  presented  in  a  complete  and  transparent 
manner  in  the  program's  budget? 

Purpose  of  the  question:  to  establish  whether  the 
performance-planning  and  budget-planning  processes  are 
integrated  so  that  1)  resource  allocation  decisions 
reflect  desired  performance  levels  (given  resource 
constraints)  and  2)  the  effects  of  funding  and  other 
policy  changes  on  results  are  clear. 

Elements  of  a  Yes  answer:  A  Yes  answer  should  reflect 
effective  program  budgeting  based  on  sound  levels  for  1) 
annual  and  long-term  performance  targets  and  2)  budget 
resources.  To  receive  a  Yes,  the  program  must  provide  a 
presentation  that  makes  clear  the  impact  of  funding, 
policy,  or  legislative  decisions  on  expected  performance 
and  explains  why  the  requested  performance/resource  mix 
is  appropriate.  A  program  with  budget  planning  that  is 
not  tied  to  performance  or  strategic  planning  would 

receive  a  No. 

A  Yes  answer  would  also  require  that  the  program  report 
all  direct  and  indirect  costs  needed  to  attain  the 

performance  results,  including  applicable  agency 
overhead,  retirement,  and  other  costs  that  might  be 
budgeted  elsewhere.  The  exclusion  of  minor  amounts  of 
services  provided  from  central  departmental  offices 
(e.g..  Office  of  the  Secretary)  from  program  costs  does 
not  require  a  program  to  receive  a  No  answer.  However,  a 
program  that  generates  significant  costs  -  which  might 
range  from  radioactive  waste  disposal  to  attorneys' 
salaries  -  that  must  be  addressed  by  another  program 

should  budget  for  these  costs  or,  at  a  minimum,  provide 
this  information  in  clear  display  tables  that  display  the 
full  costs  of  attaining  results. 

Eor  capital  assets  and  services  acquisition  programs,  a 
Yes  answer  requires  that,  in  addition  to  the  general 

criteria,  programs  explain  the  relationship  of  asset 
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acquisitions  to  overall  program  performance  goals  and 
would  be  able  to  identify  impacts  of  changes  on  program 
performance  (for  example,  the  effect  of  a  change  in  the 
quantity  acquired) . 

Evidence/Data:  Evidence  can  include  documentation  of  how 
the  budget  request  directly  supports  achieving 
performance  targets.  Budget  documents  should  also  clearly 
indicate  the  full  costs  of  achieving  performance  goals, 
even  if  some  of  these  costs  do  not  appear  in  the  specific 
account  or  activity  line  of  the  program. 

Also,  evidence  can  include  an  agency  program  budget 
estimate  that  identifies  all  spending  categories  in 
sufficient  detail  to  demonstrate  that  all  relevant  costs 
are  included  or  a  report  that  shows  the  allocation  of  all 
significant  program  overhead  costs  to  the  program. 

2.8.  Has  the  program  taken  meaningful  steps  to  correct  its 
strategic  planning  deficiencies? 

Purpose  of  the  question:  to  determine  whether  the  program 
is  on  track  to  correct  any  strategic  planning 
deficiencies  that  have  been  identified. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  that 
the  program  has  acted  to  correct  strategic  planning 
deficiencies.  A  program  that  does  not  review  planning 
efforts  or  does  not  make  corrections  to  eliminate 
identified  deficiencies  would  receive  a  No.  The  question 
addresses  any  deficiencies  identified  in  this  section. 
Particular  emphasis,  however,  should  be  placed  on  whether 
the  program  is  working  to  adopt  a  limited  number  of 
specific,  ambitious  long-term  performance  goals  and  a 
limited  number  of  annual  performance  goals  that 
demonstrate  progress  toward  achieving  the  long-term 
goals,  if  they  do  not  already  have  these  measures  or 
associated  baselines,  targets,  and  timeframes. 

For  Capital  Assets  and  Service  Acquisition  programs  (and 
relevant  R&D  programs) ,  one  strategic  planning  deficiency 
to  be  addressed  is  if  the  program  has  in  the  past 
received  a  No  to  Question  4 . CAl . 

Evidence/Data:  Evidence  can  include  a  description  of  how 
deficiencies  in  the  strategic  planning  of  a  program  are 
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identified  and  corrected,  as  well  as  examples  of  such 
changes . 


Specific  Strategic  Planning  Questions  by  Program  Type 


Regulatory  Based  Programs 

2.RG1.  Are  all  regulations  issued  by  the  program/agency 
necessary  to  meet  the  stated  goals  of  the  program,  and  do 
all  regulations  clearly  indicate  how  the  rules  contribute 
to  achievement  of  the  goals? 

Purpose  of  the  question:  to  determine  whether  the 
program  had  developed  regulations  with  clearly 
specified  goal (s) .  It  should  be  determined  whether  (1) 
the  program  is  only  issuing  those  rules  absolutely 
necessary  to  achieve  long-term  program  goals  and  is 
not  over-regulating,  (2)  all  of  the  rules  necessary  to 
meet  the  program  goals  have  been  issued,  and  (3)  the 
regulations  clearly  indicate  how  they  help  to  meet  the 
program  goals. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require 
that  only  those  regulations  that  are  absolutely 
necessary  to  accomplish  the  program  mission  and  goals 
are  promulgated  or  are  in  the  process  of  being 
promulgated;  this  determination  should  be  based  on  the 
standards  laid  out  in  Executive  Order  12866  on  the 
identification  of  the  need  for  the  regulation  and  the 
identification  of  the  market  failure  (if  applicable) . 
Additionally,  the  public  should  be  able  to  understand 
how  the  regulations  fit  into  the  overall  achievement 
of  the  program  goals.  A  Yes  response  indicates  that 
there  are  no  superfluous  regulations,  that  regulations 
are  planned  or  in  the  process  of  being  promulgated  to 
cover  regulatory  gaps  where  new  regulations  are 
required  to  accomplish  program  goals,  and  that  the 
Preamble  of  each  program  regulation  indicates  how  the 
rule  contributes  to  the  achievement  of  specific 
program  goals. 

A  program  would  receive  a  No  if  it  has  1)  obvious 
regulatory  gaps  or  outdated  regulations  in  effect  and 
2)  not  initiated  planned  actions  to  rectify  these 
problems  in  a  timely  manner. 
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Evidence/Data:  Evidence  can  include  legislation  that 
indicates  specifically  or  generically  what  regulations 
need  to  be  promulgated  as  well  as  the  rules 
themselves,  especially  the  preambles.  It  can  also 
include  internal  agency  guidance  indicating  a  process 
is  in  place  to  ensure  rulemaking  involves  a  clear 
linkage  of  the  need  for  the  rule  to  a  stated  goal.  In 
this  case,  the  agency  should  be  able  to  clearly 
articulate  this  association. 

NOTE:  Questions  2.RG1,  3.RG3,  and  4.RG1  address  the 
progression  of  reviews  for  a  program's  rules. 
Specifically,  2.RG1  addresses  targeted  development  of 
regulations , 

3.RG3  addresses  effective  implementation  of  regulations, 
and  4.RG1  addresses  maximization  of  potential  benefits 
during  the  regulation's  implementation. 


Capital  Assets  and  Service  Acquisition  Programs 


R&D  programs  addressing  the  acquisition,  construction 
or  operation  of  facilities  or  other  capital  assets 
should  answer  the  Capital  Assets  and  Service 
Acquisition  question  2.CA1). 

2 . CAl .  Has  the  agency/program  conducted  a  recent, 
meaningful,  credible  analysis  of  alternatives  that 
includes  trade-offs  between  cost,  schedule,  risk,  and 
performance  goals  and  used  the  results  to  guide  the 
resulting  activity? 

Purpose  of  the  question:  to  determine  whether  the 
agency  is  investing  in  an  asset  or  service  that 
provides  the  best  value  to  the  government. 

Elements  of  a  Yes  answer:  To  receive  a  Yes  response, 
the  agency  should  have  conducted  analyses  of 
alternatives  and  use  those  analyses.  Each  analysis 
should  include  the  baseline  assessment /status  quo, 
non-material  solutions  (e.g.,  data  compression  in  lieu 
of  a  new  data  cable) ,  consideration  of  alternatives 
(including  alternatives  to  capital  assets  and  benefit- 
cost  analysis  in  accordance  with  0MB  Circular  A-94), 
and  trade-offs  between  cost,  schedule,  and  performance 
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goals.  The  program  should  be  able  to  demonstrate  that 
the  analysis  is  credible  (e.g.,  by  having  it  reviewed 
and  validated  by  an  independent  entity  outside  the 
program) .  If  an  independent  entity's  analysis  differs 
from  the  program's  analysis,  the  program  should  defend 
differences . 

Evidence/Data:  Evidence  can  include  a  summary  of  the 
analysis  of  alternatives,  and  documentation  of  any 
independent  reviews  of  the  analysis.  As  one  source  of 
data,  an  agency  capital  asset  plan  or  business  case 
(Exhibit  300)  documentation  may  also  be  used. 


Research  and  Development  Programs 

R&D  programs  addressing  the  acquisition,  construction 
or  operation  of  facilities  or  other  capital  assets 
should  answer  the  Capital  Assets  and  Service 
Acquisition  question  (2.CA1). 


2.RD1.  If  applicable,  does  the  program  assess  and  compare 
the  potential  benefits  of  efforts  within  the  program  and 
(if  relevant)  to  other  efforts  in  other  programs  that 
have  similar  goals? 

Purpose  of  the  question:  to  determine  whether 
applicable  programs  are  evaluating  their  efforts  with 
respect  to  their  relative  potential  benefits. 
Applicable  programs  include  those  applied  R&D  programs 
that  pursue  multiple  options  toward  achieving  similar 
public  benefits.  (This  question  addresses  the  first  of 
the  industry-related  R&D  criteria,  regarding  the 
articulation  of  program  benefits  (see  Appendix  A) .) 

Elements  of  a  Yes  answer:  To  receive  a  Yes  rating,  the 
program  (or  agency)  should  conduct  periodic 
comparisons  of  the  potential  benefits  of  its  proposals 
with  alternatives.  The  program  should  be  able  to 
demonstrate  that  the  analysis  is  credible  (e.g.,  by 
having  it  reviewed  and  validated  by  an  independent 
entity  outside  the  program) .  If  an  independent 
entity's  analysis  differs  from  the  program's  analysis, 
the  program  should  defend  differences. 
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Evidence/Data:  Evidence  can  include  an  analysis 
comparing  proposed  approaches  with  alternative 
strategies.  0MB  will  work  with  agencies  as  needed  to 
assist  in  the  content  and  structure  of  these 
assessments . 


2.RD2.  Does  the  program  use  a  prioritization  process  to 
guide  budget  requests  and  funding  decisions? 

Purpose  of  the  question:  to  determine  whether  the 
program  has  clear  priorities  and  uses  them  in  budget 
requests  and  funding  decisions.  (This  question 
addresses  the  R&D  "relevance"  criterion  (see  Appendix 
A)  .) 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  a 
documented  process  to  identify  priorities  and  use 
them,  as  well  as  an  identified  set  of  current 
priorities  among  program  goals,  objectives,  and 
activities . 

R&D  programs  are  encouraged  to  work  with  independent 
advisory  bodies  to  help  prioritize  in  ways  that 
benefit  the  larger  science  and  technology  enterprise. 

Evidence/Data:  Evidence  can  include  clear  statements 
of  program  priorities  in  program  documentation  or 
mission  statements,  as  well  as  documentation  of  the 
priorities  identified  by  any  qualified  independent 
advisory  bodies. 


III.  PROGRAM  MANAGEMENT 

This  section  focuses  on  a  variety  of  elements  related  to 
whether  the  program  is  effectively  managed  to  meet  program 
performance  goals.  Key  areas  include  financial  oversight, 
evaluation  of  program  improvements,  performance  data 
collection,  and  program  manager  accountability. 
Additionally,  specific  areas  of  importance  for  each  program 
type  are  also  explored.  Potential  source  documents  and 
evidence  for  answering  questions  in  this  section  include 
financial  statements,  GAO  reports,  IG  reports,  performance 
plans,  budget  execution  data,  IT  plans,  and  independent 
program  evaluations. 
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Options  for  answers  are  Yes,  No  or  Not  Applicable.  (For  R&D 
programs,  some  of  the  questions  in  this  section  help 
address  the  prospective  aspects  of  program  "quality"  and 
"performance"  of  the  R&D  Investment  Criteria,  in  addition 
to  addressing  general  program  management  issues  (see 
Appendix  A)  .  ) 

3.1.  Does  the  agency  regularly  collect  timely  and  credible 
performance  information,  including  information  from  key 
program  partners,  and  use  it  to  manage  the  program  and 
improve  performance? 

Purpose  of  the  question:  to  determine  whether  the  program 
collects  data  on  performance  and  the  performance  of  its 
partners  and  uses  the  data  to  inform  program  management, 
resource  decisions,  and  program  performance. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  that 
the  program  agency  regularly  collect  high-quality 
performance  data  relating  to  key  program  goals  and  use 
that  information  to  adjust  program  priorities,  allocate 
resources,  or  take  other  appropriate  management  actions. 
When  key  program  activities  are  carried  out  by  other 
entities,  such  as  grantees,  agencies  should  also  consider 
their  performance  as  well.  A  Yes  also  requires  that  the 
agency  has  collected  the  baseline  performance  data 
necessary  to  set  meaningful,  ambitious  performance 
targets . 

Program  partners  are  other  agencies  or  intermediaries 
responsible  for  carrying  out  different  aspects  of  the 
program  and  might  include  partner  agencies,  grant 
recipients,  participating  financial  institutions, 
regulated  bodies,  and  contractors.  Timely  performance 
information  is  information  that  reflects  current 
performance  and  is  current  enough  to  be  useful  in  program 
management.  Credible  performance  information  is 
information  that  is  collected  through  a  systematic 
process  with  quality  controls  to  confirm  the  validity  of 
the  data. 

For  capital  assets  and  service  acquisition  programs,  a 
consideration  is  whether  the  program  uses  an  earned  value 
management  system  or  similar  system. 

For  credit  programs,  consider  whether  the  agency  or 
program  managers  regularly  collect  and  update  loan 
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performance  information  to  effectively  and  consistently 
manage  the  portfolios. 

Some  long-term  basic  research  programs  may  not  be  able  to 
define  meaningful  annual  outcome  performance  measures, 
aside  from  process  measures.  In  such  cases,  these 
programs  may  use  process-related  measures,  especially 
those  that  can  be  conceptually  linked  to  long-term 
research  goals. 

Evidence/Data:  Evidence  can  include  a  description  of  how 
the  agency  uses  performance  information  in  managing  the 
program,  as  well  as  illustrative  examples  of  recent 
management  actions  based  on  performance  information. 
Evidence  can  also  include  steps  taken  by  a  program  to 
enact  necessary  improvements  cited  by  a  specific 
evaluation . 

3.2  Are  federal  managers  and  program  partners  (including 
grantees,  sub-grantees,  contractors,  cost-sharing  partners, 
and  other  government  partners)  held  accountable  for  cost, 
schedule  and  performance  results? 

Purpose  of  the  question:  to  determine  whether  the  program 
managers  and  partners  are  accountable  for  achieving 
program  results. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  that 
the  program  agency  identify  the  managers  who  are 
responsible  for  achieving  key  program  results  and 
establish  performance  standards  for  those  managers.  When 
program  partners  contribute  to  the  achievement  of  program 
goals,  a  Yes  would  also  require  those  partners  to  achieve 
specific  performance  standards. 

In  the  case  of  block  and  formula  grant  programs,  elements 
of  a  Yes  are  not  confined  to  complying  with  the  law. 
Elements  of  a  Yes  can  include  the  presence  of  incentives 
for  managers  and  program  partners  that  would  encourage 
corrections  in  deficient  programs.  Eor  block-grant 
programs  which  support  a  wide  range  of  purposes  and  allow 
grantees  to  set  their  own  program  priorities,  this 
question  should  be  interpreted  as  whether  the  grantees 
hold  managers  accountable  and  encourage  corrections  in 
deficient  programs. 
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For  capital  assets  and  service  acquisition  programs,  a 
consideration  is  whether  contracts  include  minimum 
performance  thresholds,  incentives  for  good  performance, 
or  other  mechanisms  to  increase  accountability. 

Some  long-term  basic  research  programs  may  not  be  able  to 
define  meaningful  annual  outcome  performance  measures, 
aside  from  process  measures.  In  such  cases,  these 
programs  may  use  process-related  measures,  especially 
those  that  can  be  conceptually  linked  to  long-term 
research  goals. 

Evidence/Data:  Evidence  can  include  the  use  of 
performance  management  contracts  with  program  managers, 
or  some  other  mechanism  for  incorporating  program 
performance  into  personnel  performance  evaluation 
criteria.  Evidence  of  partners'  accountability  can 
include  requiring  grant  and  contract  awards  and  renewals 
to  consider  past  performance. 

3.3  Are  funds  (Federal  and  partners')  obligated  in  a  timely 

manner  and  spent  for  the  intended  purpose? 

Purpose  of  the  question:  to  determine  whether  funds  are 
administered  efficiently  and  obligated  in  accordance  with 
planned  schedules  and  spent  for  the  intended  purposes. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  that 
the  program  funds  be  obligated  consistently  with  the 
overall  program  plan  and  that  a  limited  amount  of 
unobligated  funds  remain  at  the  end  of  the  year.  A  Yes 
answer  would  also  require  that  programs  and  partners 
establish  schedules  for  obligations  that  properly 
correspond  to  the  resource  needs  of  the  program  plan.  In 
addition,  a  Yes  answer  requires  that  adequate  procedures 
exist  for  reporting  actual  expenditures,  comparing  them 
against  the  intended  use,  and  taking  timely  and 
appropriate  action  to  correct  single  audit  findings  when 
funds  are  not  spent  as  intended. 

A  program  would  receive  a  No  if  it  had  significant 
erroneous  payments  or  was  in  violation  of  the  Anti- 
Deficiency  Act. 


Evidence/Data:  Evidence  can  include  periodic  and  year-end 
spending  reports  from  the  program  and  its  partners. 
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Evidence  on  expenditures  can  include  spending  reports 
that  draw  intended  purpose  from  the  Congressional 
Justifications,  Appropriations,  and  program  operating 
plans  and  match  them  against  actual  spending.  For 
grantees,  evidence  can  include  grantee  audit  reports 
under  the  Single  Audit  Act,  including  data  captured  in 
the  Federal  Audit  Clearinghouse,  and  the  existence  of  an 
established  procedure  for  reviewing  actual  expenditures 
against  budgets  in  grant  awards  or  appropriate  Federal 
guidelines . 

A  No  answer  is  strong  evidence  that  a  No  may  be  warranted 
for  Question  3.6,  which  examines  financial  management 
more  generally. 

3.4.  Does  the  program  have  procedures  (e.g.,  competitive 
sourcing/cost  comparisons,  IT  improvements,  appropriate 
incentives)  to  measure  and  achieve  efficiencies  and  cost 
effectiveness  in  program  execution? 

Purpose  of  the  question:  to  determine  whether  the  program 
has  effective  management  procedures  in  place  to  ensure 
the  most  efficient  use  of  each  dollar  spent  on  program 
execution . 

Elements  of  a  Yes  answer:  The  answer  to  this  question 
should  focus  on  whether  the  program  has  regular 
procedures  in  place  to  measure  and  achieve  efficiencies 
and  cost  effectiveness,  recognizing  that  various  types  of 
procedures  may  be  acceptable.  A  Yes  would  require  that 
the  program' s  performance  plans  include  efficiency 
measures  and  targets,  such  as  per-unit  cost  of  outputs, 
timing  targets,  and  other  indicators  of  efficient  and 
productive  processes  germane  to  the  program.  In  addition, 
there  are  other  acceptable  ways  to  demonstrate  that  a 
program  has  established  procedures  for  measuring  and 
achieving  efficiencies  and  cost  effectiveness.  For 
example,  a  program  that  regularly  uses  competitive 
sourcing  to  determine  the  best  value  for  the  taxpayer, 
invests  in  IT  with  clear  goals  of  improving  efficiency, 
etc.,  could  receive  a  Yes.  A  de-layered  management 
structure  that  empowers  front  line  managers  and  that  has 
undergone  competitive  sourcing  (if  necessary)  would  also 
contribute  to  a  Yes  answer.  For  mandatory  programs,  a  Yes 
could  require  the  program  to  seek  policies  (e.g.,  through 
review  of  proposals  from  States)  that  would  reduce  unit 
costs.  Also  consider  if,  where  possible,  there  is  cross- 
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program  and  inter-agency  coordination  on  IT  issues  to 
avoid  redundancies. 

For  R&D  programs,  efficiency  measures  can  include  program 
overhead  costs,  average  times  to  fund  competitive  awards, 
etc . 

Evidence/Data:  Evidence  can  include  efficiency  measures, 
competitive-sourcing  plans,  and  IT  improvement  plans 
designed  to  produce  tangible  productivity  and  efficiency 
gains,  or  IT  business  cases  that  document  how  particular 
projects  improve  efficiency. 

3.5.  Does  the  program  collaborate  and  coordinate 

effectively  with  related  programs? 

Purpose  of  the  question:  to  determine  whether  a  Federal 
program  collaborates  with  other  related  program (s)  in  a 
meaningful  way. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  that 
the  program  collaborate  with  related  federal  programs 
and,  to  the  extent  appropriate  or  possible,  with  related 
State,  local,  and  private  programs.  A  Yes  answer  would 
require  that  the  program  show  evidence  of  collaboration 
leading  to  meaningful  actions  in  management  and  resource 
allocation.  For  example,  the  existence  of  a  coordinating 
council  would  not  by  itself  constitute  meaningful 
collaboration.  (This  question  applies  to  programs  that 
have  interrelated,  but  separately  budgeted,  efforts.  An 
example  of  an  interrelated  Federal  program  is  the  shared 
effort  of  the  Department  of  Veterans  Affairs  and  the 
Medicare  Program  to  provide  care  for  aging  veterans.) 
Meetings,  discussions  groups,  and  task  forces  are  not 
sufficient  for  a  Yes.  A  Yes  requires  evidence  of 
collaboration  leading  to  meaningful  actions  in  management 
and  resource  allocation. 

Evidence/Data:  Evidence  of  meaningful  collaboration  could 
include  joint  grant  announcements,  planning  documents, 
performance  goals,  or  referral  systems. 

3.6.  Does  the  program  use  strong  financial  management 

practices  ? 
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Purpose  of  the  question:  to  determine  whether  the  program 
uses  effective  financial  management  practices  in 
administering  program  funds. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require 
that,  at  a  minimum,  the  program  be  free  of  material 
internal  control  weaknesses  reported  by  auditors.  In 
addition,  depending  on  the  particular  risks  inherent  to 
the  program,  a  Yes  may  require  meeting  some  or  all  of  the 
following  criteria: 

•  The  program  has  procedures  in  place  to  ensure  that 
payments  are  made  properly  for  the  intended  purpose  to 
minimize  erroneous  payments. 

•  Financial  management  systems  meet  statutory 

requirements . 

•  Financial  information  is  accurate  and  timely. 

•  Integrated  financial  and  performance  systems  support 
day-to-day  operations. 

•  Financial  statements  receive  a  clean  audit  opinion 
and  have  no  material  internal  control  weaknesses. 

If  an  agency-wide  material  weakness  exists  that  is 
unrelated  to  the  program,  then  a  Yes  response  would  be 
appropriate.  However,  if  an  agency-wide  material  weakness 
has  a  direct  relation  to  the  program  (e.g.,  a  lack  of 
systems  that  support  day-to-day  operations) ,  then  the 
program  would  receive  a  No  for  this  question. 

For  block-grant  programs  which  support  a  wide  range  of 
purposes  and  allow  grantees  to  set  their  own  program 
priorities,  a  Yes  answer  should  also  require  that 
grantees  (e.g..  States  and  localities)  meet  the  criteria 
for  effective  financial  management  identified  above. 

For  credit  programs,  a  Yes  answer  would  also  require  that 
the  program  consistently  meets  the  requirements  of  the 
Federal  Credit  Reform  Act  of  1990,  the  Debt  Collection 
Improvement  Act  and  applicable  guidance  under  0MB 
Circular  A-129. 

Evidence/Data:  Evidence  can  include  recent  audit  reports 
and  existence  of  procedures  to  identify  the  above-listed 
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criteria,  such  as  the  ability  to  measure  improper 
payments . 

Unlike  Question  3.3,  the  presence  of  significant 
erroneous  payments  does  not  automatically  generate  a  No 
for  this  question.  Specifically,  if  an  agency  has 
instituted  strong  financial  management  controls  that  are 
demonstrating  measurable  improvements  in  erroneous 
payments  over  time,  then  a  Yes  may  be  appropriate. 
Nevertheless,  a  No  on  Question  3.3  is  strong  evidence 
that  a  No  may  be  appropriate  for  this  question. 

For  credit  programs,  evidence  can  include  actual  reports 
detailing  the  performance  of  the  agency' s  portfolio 
management,  subsidy  calculations,  re-estimates, 
modifications,  etc.  Other  evidence  can  include 
independent  evaluations  of  the  program's  performance. 

3.7.  Has  the  program  taken  meaningful  steps  to  address  its 
management  deficiencies? 

Purpose  of  the  question:  to  determine  whether  the  program 
has  developed  a  system  of  evaluating  program  management 
and  correcting  deficiencies  when  they  are  identified. 
This  question  should  include,  but  is  not  limited  to, 
financial  management  or  other  Presidential  Management 
Agenda  deficiencies.  However,  the  focus  of  the  question 
is  program-level  deficiencies,  as  opposed  to  agency-level 
deficiencies  that  may  not  directly  affect  the  program. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  that 
the  program  has  a  system  for  identifying  and  correcting 
program  management  deficiencies  and  uses  the  system  to 
make  necessary  corrections.  A  program  that  does  not 
review  program  management  activities  and  make  corrections 
to  eliminate  identified  deficiencies  would  receive  a  No. 

Evidence/Data:  Evidence  can  include  a  description  of  how 
deficiencies  in  the  program  management  are  identified  and 
corrected  as  well  as  examples  of  such  changes. 


Specific  Program  Management  Questions  by  Program  Type 


Competitive  Grant  Programs 
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3. col.  Are  grants  awarded  based  on  a  clear  competitive 
process  that  includes  a  qualified  assessment  of  merit? 

Purpose  of  the  question:  to  determine  whether  or  not 
grant  funds  are  distributed  according  to  a  competitive 
process  so  that  the  most  meritorious  applications  are 
awarded  and  so  that  new  applicants  of  merit  will  be 
able  to  compete  fairly  with  previous  grant  recipients. 
(For  R&D  competitive  grants  programs,  this  question  is 
central  to  addressing  the  R&D  "quality"  criterion  (see 
Appendix  A) . ) 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require 
that  the  overwhelming  majority  of  awards  (95  percent 
or  more  of  funding)  are  distributed  according  to  a 
competitive  process.  Elements  of  the  process  can 
include  independent  merit  review  and  ranking  of 
applications,  as  well  as  a  limit  on  the  percentage  of 
funds  that  are  earmarked.  A  Yes  answer  would  also 
require  that  the  program  operate  a  fair  and  open 
competition  and  provide  a  reasonable  amount  of 
outreach  to  encourage  the  participation  of  new 
grantees.  Eair  and  open  does  not  mean  that  data  should 
not  be  protected  (e.g.,  to  meet  classification  needs, 
to  protect  personal  data  or,  for  an  appropriate  span 
of  time,  intellectual  property.)  or  that  other 
reasonable  qualification  standards  can't  be  imposed 
(e.g.  meets  status  as  an  academic  organization). 
Considerations  can  include  whether  the  program  tends 
to  provide  grants  to  the  same  list  of  grantees  year 
after  year.  Awards  may  be  renewed  and  considered 
competitive  if  the  original  award  was  competitively 
awarded,  renewals  only  extend  for  a  short  and  definite 
period  of  time  and  the  possibility  of  renewal  is 
announced  in  the  original  competition.. 

Eor  R&D  programs,  the  same  competitive  hurdle  should 
apply  for  external  projects,  although  a  tightly 
defined  internal  program  can  be  maintained  if  the 
agency  can  clearly  show  that  it  is  needed  to  provide 
for  the  maintenance  of  scientific  expertise  in-house. 
Unsolicited,  potentially  breakthrough  ideas  and  out- 
of-the-box  proposals  can  be  part  of  the  portfolio  if 
they  are  merit-reviewed  from  the  beginning.  If  such 
proposals  compete  fairly  against  a  significant  number 
of  other  proposals  for  a  share  of  funding,  they  may  be 
considered  to  meet  the  competitive  standard  even  if 
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they  don't  respond  to  a  specific  request  for  proposal. 
In  addition  to  grants,  allowable  transactions  may 
include  mechanisms  such  as  contracts  or  other 
transactions.  (Merit  review  processes  and  ranking  of 
applications  should  also  apply  to  these  transaction 
vehicles . ) 


Evidence/Data:  Evidence  can  include  a  description  of 
the  awards  process,  percentage  of  funds  earmarked,  and 
percentage  of  funds  subject  to  peer  review.  Evidence 
can  also  include  the  relative  number  of  new  awardees 
per  award  cycle  and  technical  assistance  and  outreach 
efforts  of  the  agency. 


3 . C02 .  Does  the  program  have  oversight  practices  that 
provide  sufficient  knowledge  of  grantee  activities? 

Purpose  of  the  question:  to  determine  whether  or  not 
the  program  has  an  understanding  of  how  its  funds  are 
utilized  by  grantees. 


Elements  of  a  Yes  answer:  A  Yes  answer  would  require 
that  a  program  have  sufficient  oversight  capacity. 
This  capacity  may  be  demonstrated  by  a  program  that 
has  a  reporting  system  in  place  to  document  grantees' 
use  of  funds  in  eligible  activity  categories,  conducts 
site  visits  to  a  substantial  number  of  grantees  on  a 
regular  basis,  audits  grantee  performance,  and  tracks 
actual  expenditures  to  verify  that  funds  are  used  for 
their  designated  purpose.  A  program  with  a  strong 
relationship  to  its  grantees  and  a  high  level  of 
understanding  of  what  grantees  do  with  the  resources 
allocated  to  them  would  receive  a  Yes. 


A  program  with  a  poor  reporting  system  to  track 
expenditures  by  grantees  would  receive  a  No  rating. 


Eor  R&D  programs,  the  above  standards  would  apply,  but 
in  addition  to  grants  and  grantees  they  would  apply  to 
contracts,  cooperative  agreements  and  other 
transactions  and  their  awardees. 


Evidence/Data:  Evidence  can  include 

structure,  oversight  techniques,  audit 
schedule,  and/or  an  assessment  of 
quality . 


the  reporting 
or  site  visit 
program  data 
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3. cos.  Does  the  program  collect  grantee  performance  data 
on  an  annual  basis  and  make  it  available  to  the  public  in 
a  transparent  and  meaningful  manner? 

Purpose  of  the  question:  to  determine  whether  or  not 
the  program  has  a  system  in  place  to  collect  and 
present  publicly  information  that  captures  the  most 
important  impacts  of  program  performance. 

Elements  of  a  Yes  answer:  A  Yes  would  require  that  the 
program  collects,  compiles  and  disseminates  grantee 
performance  information  in  an  accessible  manner,  such 
as  via  a  web  site  or  widely  available  program  reports. 
Data  would  be  both  aggregated  on  a  program-wide  level 
and  disaggregated  at  the  grantee  level.  The  assessment 
about  the  appropriate  level  of  aggregation  of  results 
may  depend  upon  needs  to  protect  certain  data,  such  as 
classified  data,  personal  data  or,  for  a  limited  span 
of  time,  intellectual  property. 

A  program  would  receive  a  No  if  grantee  performance 
information  is  not  available  to  the  public,  or  if  it 
is  only  aggregated  at  a  high  level.  Similarly,  a 
program  could  receive  a  No  response  if  the  data  it 
presents  are  not  related  to  the  impact  of  the  program. 

For  R&D  programs,  the  above  standards  would  apply,  but 
in  addition  to  grants  and  grantees  would  apply  to 
contracts,  cooperative  agreements  and  other 
transactions  and  their  awardees. 

Evidence/Data:  Evidence  can  include  citations  of  the 
types  of  data  that  are  collected  and  disseminated  as 
well  as  a  description  of  how  these  data  are  made 
available . 


Block/Formula  Grant  Programs 

3.BF1.  Does  the  program  have  oversight  practices  that 
provide  sufficient  knowledge  of  grantee  activities? 

Purpose  of  the  question:  to  determine  whether  or  not 
the  program  has  an  understanding  of  how  its  funds  are 
utilized  by  grantees. 
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Elements  of  a  Yes  answer:  A  Yes  answer  would  require 
that  a  program  have  sufficient  oversight  capacity. 
This  capacity  may  be  demonstrated  by  a  program  that 
has  a  reporting  system  in  place  to  document  grantees' 
use  of  funds  in  eligible  activity  categories,  conducts 
site  visits  to  a  substantial  number  of  grantees  on  a 
regular  basis,  audits  grantee  performance,  and  tracks 
actual  expenditures  to  verify  that  funds  are  used  for 
their  designated  purpose.  A  program  with  a  strong 
relationship  to  its  grantees  and  a  high  level  of 
understanding  of  what  grantees  do  with  the  resources 
allocated  to  them  would  receive  a  Yes. 

A  program  with  no  reporting  system  to  track 

expenditures  by  grantees  would  receive  a  No. 

Evidence/Data:  Evidence  can  include  the  reporting 

structure,  oversight  techniques,  audit  or  site  visit 
schedule,  and/or  an  assessment  of  program  data 
quality . 

3.BE2.  Does  the  program  collect  grantee  performance  data 
on  an  annual  basis  and  make  it  available  to  the  public  in 
a  transparent  and  meaningful  manner? 

Purpose  of  the  question:  to  determine  whether  or  not 
the  program  has  a  system  in  place  to  collect  and 
present  publicly  information  that  captures  the  most 
important  impacts  of  program  performance. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require 
the  program  collects,  compiles  and  disseminates 
grantee  performance  information  in  an  accessible 
manner,  such  as  via  a  web  site  or  widely  available 
program  reports.  Data  would  be  both  aggregated  on  a 
program-wide  level  and  disaggregated  at  the  grantee 
level . 

A  program  would  receive  a  No  if  grantee  performance 
information  is  not  available  to  the  public,  or  if  it 
is  only  aggregated  at  a  high  level.  Similarly,  a 
program  could  receive  a  No  response  if  the  data  it 
presents  are  not  related  to  the  impact  of  the  program. 

Evidence/Data:  Evidence  can  include  citations  of  the 
types  of  data  that  are  collected  and  disseminated  as 
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well  as  a  description  of  how  these  data  are  made 
available . 


Regulatory  Based  Programs 

3.RG1.  Did  the  program  seek  and  take  into  account  the 
views  of  all  affected  parties  (e.g.,  consumers;  large  and 
small  businesses;  State,  local  and  tribal  governments; 
beneficiaries;  and  the  general  public)  when  developing 
significant  regulations? 

Purpose  of  the  question:  to  determine  the  level  of 
coordination,  during  the  rulemaking  process,  with 
parties  affected  by  the  regulations. 

Elements  of  a  Yes  Answer:  A  Yes  would  require  the 
program  solicits  the  opinions  of  affected  parties  on 
significant  regulations  and  thoroughly  evaluates  the 
concerns  and  suggestions  raised  by  these  entities.  For 
example,  a  program  that  seeks  the  opinions  of  affected 
parties  and  incorporates  their  suggestions  or  explains 
why  other  suggestions  were  not  incorporated  during  the 
rule  making  process  could  receive  a  Yes. 

If  the  program  drafts  its  rules  in  a  vacuum  without 
consulting  any  of  the  potentially  affected  parties,  it 
would  likely  receive  a  No.  While  the  element  of 
seeking  views  is  mandated  by  law,  the  assessment 
should  consider  the  extent  to  which  the  program  takes 
those  views  into  account. 

Evidence/Data:  Evidence  can  include  notices  seeking 
public  comment  and  addressing  comments  in  final  rules, 
regulation  preambles  which  discuss  compliance  with  the 
Regulatory  Flexibility  Act,  Unfunded  Mandates  Act  of 
1995,  Small  Business  Regulatory  Enforcement  Fairness 
Act  (SBREFA)  of  1996,  E.O.  13132,  and  National 
Environmental  Policy  Act,  and  detailed  preamble 
language  discussing  how  public  comments  were 
considered  and  addressed. 

3.RG2.  Did  the  program  prepare  adequate  regulatory  impact 
analyses  if  required  by  Executive  Order  12866,  regulatory 
flexibility  analyses  if  required  by  the  Regulatory 
Flexibility  Act  and  SBREFA,  and  cost-benefit  analyses  if 
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required  under  the  Unfunded  Mandates  Reform  Act;  and  did 
those  analyses  comply  with  0MB  guidelines? 

Purpose  of  the  question:  to  determine  whether  the 
program,  in  justifying  its  rules,  prepares  sound 
analyses  (i.e.,  cost-benefit  analysis,  risk  analysis) 
that  are  rigorous,  thorough,  and  based  upon  the  best 
available  data  and  consistent  with  0MB ' s  economic 
analysis  guidelines. 

Elements  of  a  Yes  Answer:  A  Yes  answer  should 
generally  include,  but  is  not  limited  to,  a  statement 
of  need  of  the  proposed  action,  an  examination  of 
alternative  approaches,  and  an  analysis  of  the 
incremental  benefits  and  costs  of  the  proposed  action. 
In  accordance  with  0MB' s  economic  guidelines, 
programs'  regulatory  actions  should  maximize  net 
benefits;  consider  regulatory  and  non-regulatory 
alternatives;  monetize  regulatory  costs  and  benefits; 
and  choose  the  alternative  that  maximizes  net  benefits 
in  their  regulatory  analyses.  For  example,  programs 
that  fully  document  the  impacts  on  public  health  and 
safety  and  the  regulated  industry  through  a  thorough 
benefit,  cost  and  risk  analysis  based  upon  the  best 
possible  available  data,  and  examine  other  regulatory 
and  non-regulatory  alternatives  would  receive  a  Yes.  A 
program  may  receive  a  Yes  if  its  analyses  are 
subjected  to  peer  review  by  government  entities, 
academia,  industry,  or  non-profit  research 
organizations;  however,  peer  review  is  not  required 
for  a  program  to  receive  a  Yes. 

If  a  program's  impact  analyses  fail  to  include  a 
discussion  of  the  costs  of  restrictions  on  the 
regulated  industry,  a  No  response  to  this  question 
would  be  appropriate.  If  the  program  certifies  that 
regulations  would  not  have  a  significant  economic 
impact  on  a  substantial  number  of  small  entities,  the 
program  must  provide  adequate  justification  for  that 
conclusion  to  get  a  Yes  answer. 

Evidence/Data:  Evidence  can  include  regulatory  impact 
analyses,  regulatory  flexibility  analyses,  and  cost- 
benefit  analyses  for  the  program's  rules,  any  reports 
or  feedback  generated  by  outside  reviewers,  and 
coordination  between  reviewers  and  the  sponsoring 
agency  or  program. 
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3.RG3.  Does  the  program  systematically  review  its  current 
regulations  to  ensure  consistency  among  all  regulations 
in  accomplishing  program  goals? 


Purpose  of  the  question:  to  determine  whether  the 
agency  met  the  goal  intended  when  developing  the 
regulation.  It  should  be  clear  that  the  program 
consists  of  only  those  regulations  that  are:  (1) 

necessary  in  achieving  its  goals,  (2)  relevant  to  the 
current  societal  and  economic  situation,  and  (3) 
complementary  and  consistent  with  each  other. 

Elements  of  a  Yes  Answer:  A  Yes  answer  would  require  a 
program  to  review  its  regulations  periodically  (e.g., 
every  two  years)  to  ensure  that  they  were  consistent 
with  program  policies.  A  consideration  would  include 
whether  the  program  makes  attempts  to  minimize 
regulatory  burden  through  constant  review  of 
regulations,  with  an  eye  toward  streamlining,  if 
possible.  An  additional  factor  to  consider  is  whether 
the  program  ensures  that  every  regulation  is 
consistent  with  the  program's  goals.  An  example  of  a 
Yes  could  be  a  program  that  conducts  look-back  studies 
every  third  year  on  all  of  its  significant  regulations 
to  ensure  that  they  are  all  current,  consistent,  and 
relevant  to  the  program  goals,  and,  if  the  review 
concludes  that  a  regulation  is  no  longer  necessary, 
the  program  proposes  or  takes  action  to  remedy  the 
situation . 

If  a  program,  however,  continues  to  enforce 
regulations  that  are  no  longer  justified  and/or 
necessary,  the  program  would  receive  a  No.  In 
addition,  a  program  that  has  not  reviewed  its 
regulations  for  consistency  and  necessity  in  a 
significant  time  period  would  receive  a  No. 

Evidence/Data:  Evidence  should  include: 

•  a  program  plan  or  process  to  conduct  this  exercise 
on  a  regular  basis; 

•  documentation  (such  as  assessments,  evaluations,  or 
examinations;  planned  or  completed)  generated  as  a 
result  of  the  above  processes  or  plans;  and/or 
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•  as  a  result  of  this  type  of  review,  any  changes 
made  to  or  eliminations  from  the  program  or  its 
regulations,  or  a  justification  that  no  change  is 
needed . 

NOTE:  Questions  2.RG1,  3.RG3,  and  4.RG1  address  the 

progression  of  reviews  for  a  program's  rules. 
Specifically,  2.RG1  addresses  targeted  development  of 
regulations,  3.RG3  addresses  effective  implementation 
of  regulations,  and  4.RG1  addresses  maximization  of 
potential  benefits  during  the  regulation' s 
implementation . 

3.RG4.  Are  the  regulations  designed  to  achieve  program 
goals,  to  the  extent  practicable,  by  maximizing  the  net 
benefits  of  its  regulatory  activity? 

Purpose  of  the  question:  to  determine  whether  the 
program,  as  it  promulgates  regulations,  ensures  that 
its  regulatory  requirements,  in  total,  maximize  net 
benefits.  (Note  that  this  question  relates  to  the 
promulgation  of  regulations,  as  opposed  to  their 
implementation . ) 

Elements  of  a  Yes  Answer:  A  Yes  answer  would  require 
the  program  to  maximize  overall  net  benefits  to  the 
greatest  extent  practicable.  The  program  should  show 
that  it  makes  the  best  effort  to  assess  how  each 
additional  regulation  adds  to  the  current  level  of 
regulatory  requirements  and  keeps  regulatory 
compliance  burden  at  a  minimum,  including  the  burden 
associated  with  information  collection.  Any  additional 
compliance  burdens  should  be  shown  to  increase 
benefits  substantially.  An  important  consideration  for 
this  question  is  whether  in  promulgating  its 
regulations,  for  a  given  level  of  benefits,  the  agency 
allows  alternative  methods  for  compliance,  record 
keeping,  and  reporting  to  minimize  the  cost  burden  on 
regulated  entities  (including  electronic  means) . 
Programs  should  be  judged  within  their  statutory 
framework;  the  program  should  maximize  net  benefits  to 
the  extent  allowed  by  statute  in  order  to  get  a  Yes. 

Evidence/Data:  Evidence  can  include  statistics  on 

compliance  reporting  burden  and  the  costs  of  the 
program's  requirements  on  regulated  industries  in 
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total,  or  evidence  from  a  thorough  cost-benefit 
analysis . 


Capital  Assets  and  Service  Acquisition  Programs 

3 . CAl .  Is  the  program  managed  by  maintaining  clearly 
defined  deliverables,  capability/performance 
characteristics,  and  appropriate,  credible  cost  and 
schedule  goals? 

Purpose  of  the  question:  to  determine  whether  the 
agency  has  clearly  identified  and  defined  the  required 
quality,  capability,  and  performance  characteristics 
or  objectives  expected  of  the  end  product /result  of 
the  asset  or  service  acquisition.  This  element  is 
critical  because  it  assures  that  all  parties 
(government,  contractor,  etc)  are  working  toward  the 
same  end-product  and  result. 

Elements  of  a  Yes  answer:  If  acquiring  a  capital 
asset,  a  Yes  would  require  the  program  to  validate 
planning  decisions  (procurement  phase  step  III.l  of 
the  Capital  Programming  Guide) ,  document  the 
capabilities  or  characteristics  that  are  expected,  and 
to  make  management  decisions  based  on  whether 
milestones  are  being  met.  For  example,  a  weapon  system 
that  has  defined  key  performance  parameters  and 
operational  requirements  would  get  a  Yes,  one  that  is 
proceeding  without  such  definition  should  receive  a 
No.  For  services,  a  Yes  would  require  the  program  made 
adequate  use  of  performance-based  and  fixed-priced 
type  contracts.  A  program  that  acquires  services 
through  other  than  performance-based  and  fixed-priced 
type  contracts  should  receive  a  No,  unless  there  is  a 
legitimate  reason  for  not  using  such  contracts. 

Evidence/Data:  Evidence  can  include  program 
documentation  of  planning  decision  validation  for 
asset  acquisitions,  and  documentation  describing  key 
performance  characteristics  and/or  deliverables  and 
demonstration  that  this  information  is  used 
appropriately  in  management  decisions. 
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Credit  Programs 


3 . CRl .  Is  the  program  managed  on  an  ongoing  basis  to 
assure  credit  quality  remains  sound,  collections  and 
disbursements  are  timely,  and  reporting  requirements  are 
fulfilled? 

Purpose  of  the  question:  to  determine  whether  the 
program  agency  and  its  partners  manage  the  financial 
performance  of  their  credit  programs. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require 
managing  the  program  based  on  the  results  of  an 
effective  monitoring  system  that  tracks  the  financial 
performance  of  each  credit  facility.  Collection  and 
analysis  of  borrower  repayment  streams  should  be  part 
of  the  evaluation  process  and  could  be  coupled  with 
reports  from  or  trips  to  the  field.  The  evaluation 
should  also  include  an  analysis  of  agency  efforts  to 

reduce  default  ratesT - and  maximize  collections  and 

recoveries . 

Evidence/Data:  Evidence  can  include  quarterly 

financial  statements  from  the  program,  agency. 
Treasury,  the  guaranteed  lender,  loan  servicing  agent; 
internal  evaluations,  external  independent  performance 
evaluations;  reports  from  field  representatives  or 
trips  to  the  field  on  the  borrowers'  performance. 

3 . CR2 .  Do  the  program's  credit  models  adequately  provide 
reliable,  consistent,  accurate  and  transparent  estimates 
of  costs  and  the  risk  to  the  Government? 

Purpose  of  the  questions:  to  determine  whether  the 
program  uses  a  reliable  method  for  estimating  program 
costs . 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require 
that  the  agency  employ  a  rigorous  cost-estimation 
model  that  adequately  accounts  for  the  government's 
risk  and  generates  dependable  cost  estimates  for  each 
cohort.  A  program  whose  cost  estimates  routinely 
differ  markedly  from  actual  results  should  receive  a 
No. 

Evidence/Data:  Evidence  can  include  a  description  of 
any  problems  or  advantages  of  the  program's  cost 
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estimation  model.  For  example,  a  strong  model  could 
employ  well  established  statistical  estimation 
techniques  that  have  a  solid  track  record  for 
predicting  borrower  activity.  However,  a  weaker  model 
could  fail  to  account  for  potential  contingencies  that 
increase  government  risk. 


Research  and  Development  Programs 


R&D  programs  addressing  the  acquisition,  construction  or 
operation  of  facilities  or  other  capital  assets  should 
answer  the  Capital  Assets  and  Service  Acquisition 
question  (3.CA1). 

R&D  programs  that  use  competitive  grants,  contracts, 
cooperative  agreements  or  other  transactions  should 
answer  the  Competitive  Grants  questions  (3.C01,  C02, 

C03)  . 

3.RD1.  For  R&D  programs  other  than  competitive  grants 
programs,  does  the  program  allocate  funds  and  use 
management  processes  that  maintain  program  quality? 

Purpose  of  the  question:  to  determine  whether  the 
program  uses  a  clearly  stated,  defensible  method  for 
allocating  its  R&D  funding.  This  question  is  central 
to  addressing  the  R&D  "quality"  criterion  (see 
Appendix  A) . 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require 
that  the  program  allocate  funding  using  a  broadly 
competitive  process  based  on  merit,  or  that  it  has 
compelling  justifications  for  R&D  funding  allocated 
through  other  means.  Interpretations  of  competition 
and  merit  review  should  be  consistent  with  the 
definitions  in  Circular  A-11:  "...intramural  and 

extramural  research  programs  where  funded  activities 
are  competitively  awarded  following  review  for 
scientific  and  technical  merit."  All  program  funds 
allocated  through  means  other  than  unlimited 
competition  must  document  the  processes  they  use  to 
distribute  funds  to  each  type  of  R&D  performer  (e.g., 
federal  laboratories,  federally  funded  R&D  centers, 
universities,  etc.) .  Programs  are  encouraged  to  use 
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external  assessment  of  the  methods  they  use  to 
allocate  R&D  and  maintain  program  quality. 

Evidence/Data:  Evidence  can  include  a  description  of 
the  awards  process,  percentage  of  funds  earmarked, 
percentage  of  funds  subject  to  competitive  peer 
review,  and  results  of  external  assessments. 


IV.  PROGRAM  RESULTS/ACCOUNTABILITY 

This  section  considers  whether  a  program  is  meeting  its 
long-term  and  annual  performance  goals.  This  section  also 
assesses  how  well  the  program  compares  to  similar  programs 
and  how  effective  the  program  is  based  on  independent 
evaluations.  Potential  source  documents  and  evidence  for 
answering  questions  in  this  section  include  GPRA 
performance  reports,  evaluations,  GAO  reports,  IG  reports 
and  other  agency  documents.  Assessments  of  program  results 
should  be  based  on  the  most  recent  reporting  cycle  or  other 
relevant  data.  The  Measures  tab  in  the  PART  worksheet 
contains  data  fields  for  a  performance  targets  and  results, 
and  should  be  completed  to  the  greatest  extent  possible  for 
all  measures  agreed  to  by  0MB  and  the  agency  for  Section 
II . 

Answers  in  this  section  are  rated  as  Yes,  Large  Extent, 
Small  Extent,  and  No.  Like  Sections  I-III,  the  scoring 
system  in  this  section  remains  on  a  0  to  1  point  scale. 
Scoring  for  this  section  differs  by  including  the  option  of 
partial  credit  between  0  and  1  in  increments  of  0,  .33, 

.67,  and  1. 

In  general.  Not  Applicable  answers  are  not  appropriate  for 
Questions  4.1  and  4.2.  While  it  is  recognized  that  some 
programs  may  have  great  difficulty  developing  quantitative 
performance  goals,  programs  are  strongly  encouraged  to  have 
some  meaningful  and  appropriate  methods  for  demonstrating 
results.  0MB  and  agencies  should  work  together  to  develop 
approaches  for  programs  where  it  is  difficult  to  develop 
quantitative  measures,  and  where  qualitative,  expert- 
review,  or  other  measures  are  more  appropriate.  Not 
Applicable  is  a  potential  answer  for  Question  4.3  if  the 
program  is  already  operating  at  very  high  efficiency 
levels.  It  may  also  be  a  possible  answer  for  Question  4.4 
and,  with  specific  justification,  for  Question  4.5.  (For 
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R&D  programs,  most  of  the  questions  in  this  section  help 
address  the  retrospective  aspects  of  the  R&D  Investment 
Criteria,  with  emphasis  on  the  "performance"  criterion  (see 
Appendix  A) . ) 

4.1.  Has  the  program  demonstrated  adequate  progress  in 
achieving  its  long-term  performance  goals? 

Purpose  of  the  question:  to  determine  whether  the  program 
is  meeting  or  making  measurable  progress  toward  meeting 

the  long-term  performance  goals  evaluated  in  Questions 

2.1  and  2.2.  The  question  also  seeks  to  determine  whether 
the  program's  partners  are  meeting  long-term  goals 
evaluated  in  Question  2.5,  if  partner  performance  is 
critical  to  the  program  achieving  its  goals.  Examples  of 
partners  can  include  grantees,  participating  financial 
institutions,  regulated  bodies,  or  suppliers. 

Elements  of  a  Yes  answer:  A  Yes  answer  (i.e.,  full 
credit)  would  require  that  the  program  is  on  track  to 

meet  all  the  long-term  performance  goals  -  including 

ambitious  targets  and  timeframes  -  evaluated  in  Questions 
2.1  and  2.2.  A  program  would  not  receive  a  Yes  answer  by 
simply  meeting  any  one  of  its  long-term  targets,  or  by 
having  performance  measures  but  no  ambitious  targets  and 
timeframes.  A  Yes  answer  would  also  require  that,  where 
applicable,  partners  commit  to  long-term  outcome  targets 
and  achieve  them  as  well.  Where  relevant,  a  Yes  answer 
would  also  require  that  a  program  has  addressed 
appropriately  any  predefined  end  targets. 

Partial  credit,  such  as  Large  Extent  or  Small  Extent, 
should  be  given  in  cases  where  there  is  partial,  but 
notable,  achievement  of  long-term  targets.  A  program 
could  receive  a  No  if  it  had  received  a  Yes  for  achieving 
its  annual  targets  (next  question) ,  but  is  not  making 
progress  toward  meeting  its  long-term  goals. 

Additional  rating  guidance: 

•  If  adequate  outcome  (or  output)  measures  are  not 
available  and  a  program  received  a  No  in  Question  2.1, 
the  program  must  receive  a  No  answer  to  this  question. 

•  If  the  program  received  a  Yes  in  Question  2.1  and  a 
No  in  Question  2.2,  then  the  program  cannot  receive  a 
rating  higher  than  Small  Extent. 
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The  only  exceptions  to  this  guidance  are  in  cases  in 
which  0MB  has  approved  the  use  of  alternative  forms  of 
assessment,  as  discussed  in  the  Section  IV  overview. 

Evidence/Data:  Evidence  can  include  data  from  the 
agency's  GPRA  performance  report,  a  strategic  plan,  or 
other  Administration  goals  and  objectives.  Reports 
detailing  customer  satisfaction  with  program  performance, 
program  reports  detailing  rates  of  utilization  or 
participation,  or  independent  evaluations  of  the 
program' s  performance  may  also  be  considered  as  relevant 
evidence.  In  cases  where  targets  are  not  met,  additional 
evidence  can  include  an  explanation  of  the  main  reasons. 

Space  is  provided  in  the  Measures  tab  of  the  PART 
worksheet  to  list  and  document  goals,  targets  and 
achieved  results.  Only  measures  that  meet  the  standards 
for  a  Yes  should  be  entered  on  the  worksheet. 

4.2.  Does  the  program  (including  program  partners)  achieve 

its  annual  performance  goals? 

Purpose  of  the  question:  to  determine  whether  the  program 
is  meeting  the  targets  evaluated  in  Question  2.4.  The 
question  also  seeks  to  determine  whether  the  program's 
partners  are  meeting  annual  targets  evaluated  in  Question 
2.5,  if  partner  performance  is  critical  to  the  program 
achieving  its  overall  targets.  Examples  of  partners  can 
include  grantees,  contractors,  participating  financial 
institutions,  regulated  bodies,  or  suppliers. 

Elements  of  a  Yes  answer:  A  Yes  answer  (i.e.,  full 
credit)  would  require  that  the  program  meet  all  the 
annual  performance  targets  evaluated  in  Question  2.4.  A 
Yes  answer  would  also  require  the  program  received  a  Yes 
for  Questions  2.1  and  2.3,  and  a  Yes  or  Not  Applicable 
for  Question  2.5.  A  program  would  not  receive  a  Yes 
answer  by  simply  meeting  any  one  of  its  annual  targets.  A 
Yes  answer  would  also  require  that,  where  applicable, 
partners  commit  to  annual  targets  and  achieve  them  as 
well . 

Partial  credit  such  as  Large  Extent  or  Small  Extent, 
should  be  given  in  cases  where  there  is  partial,  but 
notable,  achievement  of  targets. 
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Additional  rating  guidance: 


•  If  a  program  received  a  No  in  Question  2.3,  the 
program  must  receive  a  No  answer  to  this  question. 

•  If  the  program  received  a  Yes  in  Question  2.3  and  a 
No  in  Question  2.4,  then  the  program  cannot  receive  a 
rating  higher  than  Small  Extent. 

The  only  exceptions  to  this  guidance  are  in  cases  in 
which  0MB  has  approved  the  use  of  alternative  forms  of 
assessment,  as  discussed  in  the  Section  IV  overview. 

Evidence/Data:  Evidence  can  include  data  from  the 

agency's  annual  GPRA  performance  report,  a  strategic 
plan,  or  other  Administration  goals  and  objectives.  In 
cases  where  targets  are  not  met,  additional  evidence  can 
include  an  explanation  of  the  main  reasons. 

Space  is  provided  in  the  Measures  tab  of  the  PART 
worksheet  to  list  and  document  goals,  targets  and 
achieved  results.  Only  measures  that  meet  the  standards 
for  a  Yes  should  be  entered  on  the  worksheet. 

4.3.  Does  the  program  demonstrate  improved  efficiencies  or 
cost  effectiveness  in  achieving  program  goals  each  year? 

Purpose  of  the  question:  to  determine  whether  management 
practices  have  resulted  in  efficiency  gains  over  the  past 
year . 

Elements  of  a  Yes  answer:  A  Yes  would  require  that  the 
program  demonstrate  improved  efficiency  or  cost 
effectiveness  over  the  prior  year.  Efficiency 

improvements  should  generally  be  measured  in  terms  of 
dollars  or  time.  Programs  that  complete  an  A-76 

competition  -  an  indicator  of  cost-efficient  processes  - 
would  also  likely  be  eligible  for  a  Yes  answer,  provided 
that  the  competition  addresses  the  program' s  key  cost  and 
performance  drivers.  Also,  programs  that  clearly 
demonstrate  very  high  levels  of  efficiency  through  other 
means  may  receive  a  Yes  without  documenting  increasing 
efficiency  over  time.  A  program  would  normally  not  be 
eligible  for  a  Yes  answer  to  this  question  if  it  received 
a  No  in  Question  3.4. 
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Evidence/Data:  Evidence  can  include  meeting  performance 
targets  to  reduce  per  unit  costs  or  time,  meeting 
production  and  schedule  targets;  or  meeting  other  targets 
that  result  in  tangible  productivity  or  efficiency  gains. 
Efficiency  measures  may  also  be  considered  in  Questions 
4 . 1  and  4.2. 

4.4.  Does  the  performance  of  this  program  compare  favorably 
to  other  programs,  including  government,  private,  etc., 
with  similar  purpose  and  goals? 

Purpose  of  the  question:  to  determine  how  well  the 
program  performs  relative  to  other  programs  engaged  in  a 
similar  activity. 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  the 
program  compare  favorably  to  other  programs  with  similar 
purpose  and  goals.  Programs  are  not  limited  to  Eederal 
government  and  can  include  State  and  local  government  and 
the  private  sector.  The  user  should  consider  relevant 
evaluations  that  allow  a  comparison  of  programs  with 
similar  purpose  and  goals.  A  Not  Applicable  rating  is 
appropriate  if  1)  no  comparable  federal,  state,  local 
government,  or  private  sector  programs  exist,  or  2)  the 
comparison  would  be  too  inherently  difficult  and  costly 
to  perform  for  the  foreseeable  future.  The  explanation 
for  Not  Applicable  should  explain  why  the  comparison  is 
inherently  too  difficult  to  perform.  (This  question  is 
not  limited  to  comparisons  of  programs  with  explicitly 
coordinated  "common  measures.") 

For  capital  assets  and  service  acquisition  programs, 
review  of  performance  should  include  cost /schedule 
adherence,  quality,  and  quantity  of  deliverables. 

Evidence/Data:  Evidence  can  include  evaluations  and 
documentation  comparing  similar  programs. 

4.5.  Do  independent  evaluations  of  sufficient  scope  and 
quality  indicate  that  the  program  is  effective  and 
achieving  results? 

Purpose  of  the  question:  to  determine  whether  the  program 
is  effective  based  on  independent  and  comprehensive 
evaluations.  This  question  may  be  particularly  important 
for  programs  that  have  substantial  difficulty  formulating 
quantitative  performance  measures.  (For  R&D  programs. 
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this  question  is  central  to  retrospective  assessment  of 
all  of  the  R&D  criteria  (see  Appendix  A) .) 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require  that 
independent  program  evaluations  indicate  that  the  program 
is  effective.  (If  the  evaluations  of  Question  2.6  address 
past  effectiveness  and  accomplishments,  they  may  be  used 
as  evidence  for  this  question  as  well.)  The  quality  of 
evaluations  of  the  program  presented  in  Question  2 . 6 
should  strongly  be  considered  in  answering  this  question. 
The  most  definitive  data  supporting  a  program' s  overall 
effectiveness  would  be  from  a  randomized  controlled 
trial,  when  appropriate  and  feasible.  Data  from  other 
evaluation  methods,  such  as  quasi-experimental  and  non- 
experimental ,  can  be  considered  as  detailed  in  Question 
2.6,  but  should  be  scrutinized  given  the  increased 
possibility  of  an  erroneous  conclusion.  If  a  program  is 
taking  necessary  steps  to  correct  deficiencies  uncovered 
by  the  evaluation,  the  user  should  address  this  effort  in 
Question  3.7. 

Relevant  evaluations  would  be  at  the  national  program 
level,  rather  than  evaluations  of  one  or  more  program 
partners,  and  would  not  focus  only  on  process  indicators 
such  as  the  number  of  grants  provided,  or  hits  on  a  web 
site.  Relevant  evaluations  would  consider  a  program's 
impact  and  effectiveness.  Evaluations  conducted  by  the 
program  itself  should  not  be  considered  "independent." 
However,  if  the  program  has  contracted  out  the  evaluation 
to  a  third-party,  it  might  be  considered  independent. 
Evaluations  conducted  by  an  agency' s  Inspector  General  or 
program-evaluation  office  also  might  be  considered 
"independent."  0MB  examiners  and  agency  staff  will 
determine  if  a  specific  evaluation  can  be  considered 
"independent"  for  this  question. 

Not  Applicable  is  not  an  option  for  this  question;  given 
the  flexibility  in  determining  what  constitutes  an 
evaluation,  all  programs  should  undergo  an  evaluation 
that  meets  the  elements  of  quality,  scope,  and 
independence  detailed  in  Question  2.6. 

Evidence/Data:  Evidence  can  include  findings  of  an 
evaluation  conducted  by  the  General  Accounting  Office, 
Inspectors  General,  academic  and  research  institutions, 
agency  contracts,  or  other  independent  entities. 
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Specific  Results  Questions  by  Program  Type 


Regulatory  Based  Programs 

4.RG1.  Were  programmatic  goals  (and  benefits)  achieved  at 
the  least  incremental  societal  cost  and  did  the  program 
maximize  net  benefits? 

Purpose  of  the  question:  to  determine  whether  the 
program  met  its  goals  in  the  most  efficient  way 
possible.  It  should  be  determined  whether  the  program 
maximized  net  benefits  through  implementation  of  its 
regulatory  actions  (as  opposed  to  regulatory 
development) .  In  calculating  the  incremental  costs  of 
a  new  regulation,  these  costs  should  be  compared  to  a 
baseline  or,  in  a  small  number  of  cases,  a  less 
stringent  alternative.  This  question  deals  with  the 
actual  implementation  of  the  regulatory  action,  not 
just  the  conception  and  promulgation  of  the  regulatory 
action . 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require 
that  the  program' s  implementation  of  the  regulatory 
action  maximizes  net  benefits.  For  example,  a 
Department  of  Transportation  maximum  load  regulation 
that  demonstrates  that  1)  the  realized  benefits  to 
health  and  safety  outweigh  the  realized  incremental 
costs  of  compliance,  and  2)  the  net  benefits  of  the 
regulatory  approach  adopted  are  higher  than  reasonable 
alternatives,  would  receive  a  Yes. 

If  a  program' s  regulations  result  in  greater 
incremental  costs  than  benefits,  or  if  the  program 
does  not  analyze  alternative  approaches  to  demonstrate 
that  its  regulatory  approach  maximized  net  benefits, 
the  program  should  get  a  No. 

Evidence/Data:  Evidence  can  include  Regulatory  Impact 
Analyses  (RIA)  or  other  supporting  programmatic 
analyses  (with  supporting  data  after  implementation 
that  assumptions  were  correct) ,  look-back  studies, 
independent  evaluations,  or  additional  impact  analyses 
using  retrospective  data.  If  a  No  answer  is 
attributable  to  statutory  requirements  to  regulate 
despite  the  fact  that  incremental  costs  exceed 
benefits,  these  statutory  requirements  should  be 
included  in  the  evidence  section. 
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NOTE:  Questions  2.RG1,  3.RG3,  and  4.RG1  address  the 
progression  of  review  for  a  program's  rules. 
Specifically,  2.RG1  addresses  targeted  development  of 
regulations,  3.RG3  addresses  effective  implementation 
of  regulations,  and  4.RG1  addresses  maximization  of 
potential  benefits  during  the  regulation' s 
implementation . 


Capital  Assets  and  Service  Acquisition  Programs 

4 . CAl .  Were  program  goals  achieved  within  budgeted  costs 

and  established  schedules? 

Purpose  of  the  question:  to  determine  whether  valid 
program  goals  were  achieved  within  budgeted  costs  and 
established  schedules  and  whether  the  program  spends 
funds  as  planned  and  budgeted.  For  capital  assets 
procured  and  in  operation,  this  question  also 
addresses  management-in-use  (i.e.,  the  operations  and 
disposal  costs) . 

Elements  of  a  Yes  answer:  A  Yes  answer  would  require 
that  the  program  achieved  the  goals  evaluated  in 
Section  II  on  budget  and  on  schedule.  An  example  of  a 
program  that  could  receive  a  No  rating  could  be  an 
acquisition  program  that  has  experienced  60  percent 
cost  growth  and  is  behind  schedule.  If  a  program's 
cost  and  schedule  targets  were  changed  in  the  last  12 
months  specifically  due  to  failure  to  achieve  previous 
goals,  the  program  should  get  a  No. 

Evidence/Data:  Evidence  can  include  a  comparison  of 
the  contract  schedule,  deliverables,  and  costs  with 
the  final  outcomes  for  that  fiscal  year. 


Research  and  Development  Programs 

R&D  programs  addressing  the  acquisition,  construction  or 
operation  of  facilities  or  other  capital  assets  should 
answer  the  Capital  Assets  and  Service  Acquisition 
question  (4. CAl). 
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For  Research  and  Development  Programs:  R&D  Investment 
Criteria 

As  another  initiative  of  the  President's  Management 
Agenda,  the  development  of  explicit  R&D  investment 
criteria  builds  on  the  best  of  the  planning  and 
assessment  practices  that  R&D  program  managers  use  to 
plan  and  assess  their  programs.  The  Administration  has 
worked  with  experts  and  stakeholders  over  the  past  two 
years  to  build  upon  lessons  learned  from  previous 
approaches . 

This  year,  the  content  of  the  R&D  investment  criteria  has 
not  changed  from  last  year,  but  0MB  has  worked  to  clarify 
its  implementation.  Specifically,  agencies  should  use  the 
criteria  as  broad  guidelines  that  apply  at  all  levels  of 
federally  funded  R&D  efforts,  and  they  should  use  the 
PART  as  the  instrument  to  periodically  evaluate 
compliance  with  the  criteria  at  the  program  level.  To 
make  this  possible,  the  R&D  PART  has  been  modified  to 
clarify  its  alignment  with  the  R&D  criteria.  The  R&D 
criteria  are  reprinted  here  as  a  guiding  framework  for 
addressing  the  R&D  PART.  More  details  will  follow  on 
other  aspects  of  R&D  criteria  implementation  this  year. 

The  R&D  criteria  address  not  only  planning,  management, 
and  prospective  assessment  but  also  retrospective 
assessment.  Retrospective  review  of  whether  investments 
were  well-directed,  efficient,  and  productive  is 
essential  for  validating  program  design  and  instilling 
confidence  that  future  investments  will  be  wisely 
invested.  Retrospective  reviews  should  address  continuing 
program  relevance,  quality,  and  successful  performance  to 
date . 

While  the  criteria  are  intended  to  apply  to  all  types  of 
R&D,  the  Administration  is  aware  that  predicting  and 
assessing  the  outcomes  of  basic  research  in  particular  is 
never  easy.  Serendipitous  results  are  often  the  most 
interesting  and  ultimately  may  have  the  most  value. 
Taking  risks  and  working  toward  dif f icult-to-attain  goals 
are  important  aspects  of  good  research  management,  and 
innovation  and  breakthroughs  are  among  the  results. 
However,  there  is  no  inherent  conflict  between  these 
facts  and  a  call  for  clearer  information  about  program 
goals  and  performance  toward  achieving  those  goals.  The 
Administration  expects  agencies  to  focus  on  improving  the 
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management  of  their  research  programs  and  adopting 
effective  practices,  and  not  on  predicting  the 
unpredictable . 

The  R&D  investment  criteria  have  several  potential 
benefits : 

•  Use  of  the  criteria  allows  policy  makers  to  make 
decisions  about  programs  based  on  information  beyond 
anecdotes,  prior-year  funding  levels,  and  lobbying  of 
special  interests. 

•  A  dedicated  effort  to  improve  the  process  for 
budgeting,  selecting,  and  managing  R&D  programs  is 
helping  to  increase  the  return  on  taxpayer  investment 
and  the  productivity  of  the  federal  R&D  portfolio. 

•  The  R&D  investment  criteria  will  help  communicate  the 
Administration' s  expectations  for  proper  program 
management . 

•  The  criteria  and  subsequent  implementation  guidance 
will  also  set  standards  for  information  to  be  provided 
in  program  plans  and  budget  justifications. 

•  The  processes  and  collected  information  promoted 
under  the  criteria  will  improve  public  understanding 
of  the  possible  benefits  and  effectiveness  of  the 
federal  investment  in  R&D. 

The  R&D  Investment  Criteria 

The  Relevance,  Quality,  and  Performance  criteria  apply  to 
all  R&D  programs.  Industry-  or  market-relevant  applied 
R&D  must  meet  additional  criteria.  Together,  these 
criteria  can  be  used  to  assess  the  need,  relevance, 
appropriateness,  quality,  and  performance  of  federal  R&D 
programs . 

I.  Relevance 

R&D  investments  must  have  clear  plans,  must  be 
relevant  to  national  priorities,  agency  missions, 
relevant  fields,  and  "customer"  needs,  and  must 
justify  their  claim  on  taxpayer  resources.  Programs 
that  directly  support  Presidential  priorities  may 
receive  special  consideration  with  adequate 
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documentation  of  their  relevance.  Review  committees 
should  assess  program  objectives  and  goals  on  their 
relevance  to  national  needs,  "customer"  needs,  agency 
missions,  and  the  field  (s)  of  study  the  program 
strives  to  address.  For  example,  the  Joint  DOE/NSF 
Nuclear  Sciences  Advisory  Committee's  Long  Range  Plan 
and  the  Astronomy  Decadal  Surveys  are  the  products  of 
good  planning  processes  because  they  articulate  goals 
and  priorities  for  research  opportunities  within  and 
across  their  respective  fields. 

0MB  will  work  with  some  programs  to  identify 
quantitative  metrics  to  estimate  and  compare  potential 
benefits  across  programs  with  similar  goals.  Such 
comparisons  may  be  within  an  agency  or  among  agencies. 

A.  Programs  must  have  complete  plans,  with  clear 
goals  and  priorities. 

Programs  must  provide  complete  plans,  which  include 
explicit  statements  of: 

-  specific  issues  motivating  the  program; 

-  broad  goals  and  more  specific  tasks  meant  to 
address  the  issues; 

-  priorities  among  goals  and  activities  within 
the  program; 

-  human  and  capital  resources  anticipated;  and 

-  intended  program  outcomes,  against  which 
success  may  later  be  assessed. 

B.  Programs  must  articulate  the  potential  public 
benefits  of  the  program. 

Programs  must  identify  potential  benefits,  including 
added  benefits  beyond  those  of  any  similar  efforts 
that  have  been  or  are  being  funded  by  the  government 
or  others.  R&D  benefits  may  include  technologies  and 
methods  that  could  provide  new  options  in  the 
future,  if  the  landscape  of  today's  needs  and 
capabilities  changes  dramatically.  Some  programs  and 
sub-program  units  may  be  required  to  quantitatively 
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estimate  expected  benefits,  which  would  include 
metrics  to  permit  meaningful  comparisons  among 
programs  that  promise  similar  benefits.  While  all 
programs  should  try  to  articulate  potential 
benefits,  0MB  and  OSTP  recognize  the  difficulty  in 
predicting  the  outcomes  of  basic  research. 
Consequently,  agencies  may  be  allowed  to  relax  this 
as  a  requirement  for  basic  research  programs. 


C.  Programs  must  document  their  relevance  to  specific 
Presidential  priorities  to  receive  special 
consideration . 

Many  areas  of  research  warrant  some  level  of  federal 
funding.  Nonetheless,  the  President  has  identified  a 
few  specific  areas  of  research  that  are  particularly 
important.  To  the  extent  a  proposed  project  can 
document  how  it  directly  addresses  one  of  these 
areas,  it  may  be  given  preferential  treatment. 


D.  Program  relevance  to  the  needs  of  the  Nation,  of 
fields  of  Science  &  Technology,  and  of  program 

"customers"  must  be  assessed  through  prospective 
external  review. 

Programs  must  be  assessed  on  their  relevance  to 

agency  missions,  fields  of  science  or  technology,  or 
other  "customer"  needs.  A  customer  may  be  another 
program  at  the  same  or  another  agency,  an 

interagency  initiative  or  partnership,  or  a  firm  or 
other  organization  from  another  sector  or  country. 
As  appropriate,  programs  must  define  a  plan  for 
regular  reviews  by  primary  customers  of  the 

program's  relevance  to  their  needs.  These  programs 
must  provide  a  plan  for  addressing  the  conclusions 
of  external  reviews . 


E.  Program  relevance  to  the  needs  of  the  Nation,  of 
fields  of  S&T,  and  of  program  "customers"  must  be 
assessed  periodically  through  retrospective  external 
review . 

Programs  must  periodically  assess  the  need  for  the 
program  and  its  relevance  to  customers  against  the 
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original  justifications.  Programs  must  provide  a  plan 
for  addressing  the  conclusions  of  external  reviews. 

II.  Quality 

Programs  should  maximize  the  quality  of  the  R&D  they  fund 
through  the  use  of  a  clearly  stated,  defensible  method 
for  awarding  a  significant  majority  of  their  funding.  A 
customary  method  for  promoting  R&D  quality  is  the  use  of 
a  competitive,  merit-based  process.  NSF' s  process  for  the 
peer-reviewed,  competitive  award  of  its  R&D  grants  is  a 
good  example.  Justifications  for  processes  other  than 
competitive  merit  review  may  include  "outside-the-box" 
thinking,  a  need  for  timeliness  (e.g.,  R&D  grants  for 
rapid  response  studies  of  Pfisteria) ,  unique  skills  or 
facilities,  or  a  proven  record  of  outstanding  performance 
(e.g.,  performance-based  renewals). 

Programs  must  assess  and  report  on  the  quality  of  current 
and  past  R&D.  For  example,  NSF' s  use  of  Committees  of 
Visitors,  which  review  NSF  directorates,  is  an  example  of 
a  good  quality-assessment  tool.  0MB  and  OSTP  encourage 
agencies  to  provide  the  means  by  which  their  programs  may 
be  benchmarked  internationally  or  across  agencies,  which 
provides  one  indicator  of  program  quality. 


A.  Programs  allocating  funds  through  means  other  than  a 
competitive,  merit-based  process  must  justify  funding 
methods  and  document  how  quality  is  maintained. 

Programs  must  clearly  describe  how  much  of  the 
requested  funding  will  be  broadly  competitive  based  on 
merit,  providing  compelling  justifications  for  R&D 
funding  allocated  through  other  means.  (See  0MB 
Circular  A-11  for  definitions  of  competitive  merit 
review  and  other  means  of  allocating  federal  research 
funding.)  All  program  funds  allocated  through  means 
other  than  unlimited  competition  must  document  the 
processes  they  will  use  to  distribute  funds  to  each 
type  of  R&D  performer  (e.g.,  federal  laboratories, 
federally  funded  R&D  centers,  universities,  etc.). 
Programs  are  encouraged  to  use  external  assessment  of 
the  methods  they  use  to  allocate  R&D  and  maintain 
program  quality. 
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B.  Program  quality  must  be  assessed  periodically  through 
retrospective  expert  review. 

Programs  must  institute  a  plan  for  regular,  external 
reviews  of  the  quality  of  the  program's  research  and 
research  performers,  including  a  plan  to  use  the 
results  from  these  reviews  to  guide  future  program 
decisions.  Rolling  reviews  performed  every  3-5  years 
by  advisory  committees  can  satisfy  this  requirement. 
Benchmarking  of  scientific  leadership  and  other 
factors  provides  an  effective  means  of  assessing 
program  quality  relative  to  other  programs,  other 
agencies,  and  other  countries. 

III.  Performance 

R&D  programs  should  maintain  a  set  of  high  priority, 
multi-year  R&D  objectives  with  annual  performance  outputs 
and  milestones  that  show  how  one  or  more  outcomes  will  be 
reached.  Metrics  should  be  defined  not  only  to  encourage 
individual  program  performance  but  also  to  promote,  as 
appropriate,  broader  goals,  such  as  innovation, 
cooperation,  education,  and  dissemination  of  knowledge, 
applications,  or  tools. 

0MB  encourages  agencies  to  make  the  processes  they  use  to 
satisfy  the  Government  Performance  and  Results  Act  (GRPA) 
consistent  with  the  goals  and  metrics  they  use  to  satisfy 
these  R&D  criteria.  Satisfying  the  R&D  performance 
criteria  for  a  given  program  should  serve  to  set  and 
evaluate  R&D  performance  goals  for  the  purposes  of  GPRA. 
0MB  expects  goals  and  performance  measures  that  satisfy 
the  R&D  criteria  to  be  reflected  in  agency  performance 
plans . 

Programs  must  demonstrate  an  ability  to  manage  in  a 
manner  that  produces  identifiable  results.  At  the  same 
time,  taking  risks  and  working  toward  dif f icult-to-attain 
goals  are  important  aspects  of  good  research  management, 
especially  for  basic  research.  The  intent  of  the 
investment  criteria  is  not  to  drive  basic  research 
programs  to  pursue  less  risky  research  that  has  a  greater 
chance  of  success.  Instead,  the  Administration  will  focus 
on  improving  the  management  of  basic  research  programs. 

0MB  will  work  with  some  programs  to  identify  quantitative 
metrics  to  compare  performance  across  programs  with 
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similar  goals.  Such  comparisons  may  be  within  an  agency 
or  among  agencies. 

Construction  projects  and  facility  operations  will 
require  additional  performance  metrics.  Cost  and  schedule 
earned-value  metrics  for  the  construction  of  R&D 
facilities  must  be  tracked  and  reported.  Within  DOE,  the 
Office  of  Science's  formalized  independent  reviews  of 
technical  cost,  scope,  and  schedule  baselines  and  project 
management  of  construction  projects  ("Lehman  Reviews") 
are  widely  recognized  as  an  effective  practice  for 
discovering  and  correcting  problems  involved  with 
complex,  one-of-a-kind  construction  projects. 


A.  Programs  may  be  required  to  track  and  report  relevant 
program  inputs  annually. 

Programs  may  be  expected  to  report  relevant  program 
inputs,  which  could  include  statistics  on  overhead, 
intramural/extramural  spending,  infrastructure,  and 
human  capital.  These  inputs  should  be  discussed  with 
0MB. 


B.  Programs  must  define  appropriate  output  and  outcome 
measures,  schedules,  and  decision  points. 

Programs  must  provide  single-  and  multi-year  R&D 
objectives,  with  annual  performance  outputs,  to  track 
how  the  program  will  improve  scientific  understanding 
and  its  application.  Programs  must  provide  schedules 
with  annual  milestones  for  future  competitions, 
decisions,  and  termination  points,  highlighting 
changes  from  previous  schedules.  Program  proposals 
must  define  what  would  be  a  minimally  effective 
program  and  a  successful  program.  Agencies  should 
define  appropriate  output  and  outcome  measures  for  all 
R&D  programs,  but  agencies  should  not  expect 
fundamental  basic  research  to  be  able  to  identify 
outcomes  and  measure  performance  in  the  same  way  that 
applied  research  or  development  are  able  to. 
Highlighting  the  results  of  basic  research  is 
important,  but  it  should  not  come  at  the  expense  of 
risk-taking  and  innovation.  For  some  basic  research 
programs,  0MB  may  accept  the  use  of  qualitative 
outcome  measures  and  quantitative  process  metrics. 
Facilities  programs  must  define  metrics  and  methods 
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(e.g.,  earned-value  reporting)  to  track  development 
costs  and  to  assess  the  use  and  needs  of  operational 
facilities  over  time.  If  leadership  in  a  particular 

field  is  a  goal  for  a  program  or  agency,  0MB  and  OSTP 

encourage  the  use  of  benchmarks  to  assess  the 

processes  and  outcomes  of  the  program  with  respect  to 
leadership.  0MB  encourages  agencies  to  make  the 
processes  they  use  to  satisfy  GPRA  consistent  with  the 
goals  and  metrics  they  use  to  satisfy  these  R&D 

criteria . 


C.  Program  performance  must  be  retrospectively  documented 
annually . 

Programs  must  document  performance  against  previously 
defined  output  and  outcome  metrics,  including  progress 
toward  objectives,  decisions,  and  termination  points 
or  other  transitions.  Programs  with  similar  goals  may 
be  compared  on  the  basis  of  their  performance.  0MB 
will  work  with  agencies  to  identify  such  programs  and 
appropriate  metrics  to  enable  such  comparisons. 

IV.  Criteria  for  R&D  Programs  Developing  Technologies  That 
Address  Industry  Issues 

The  purpose  of  some  R&D  and  technology  demonstration 
programs  and  projects  is  to  introduce  some  product  or 
concept  into  the  marketplace.  However,  some  of  these 
efforts  engage  in  activities  that  industry  is  capable  of 
doing  and  may  discourage  or  even  displace  industry 
investment  that  would  occur  otherwise.  For  the  purposes 
of  assessing  federal  R&D  investments,  the  following 
criteria  should  be  used  to  assess  industry-relevant  R&D 
and  demonstration  projects,  including,  at  0MB  discretion, 
associated  construction  activities. 

0MB  will  work  with  programs  to  identify  quantitative 
metrics  to  measure  and  compare  potential  benefits  and 
performance  across  programs  with  similar  goals,  as  well 
as  ways  to  assess  market  relevance. 


A.  Programs  and  projects  must  articulate  public  benefits 
of  the  program  using  uniform  benefit  indicators  across 
programs  and  projects  with  similar  goals. 
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In  addition  to  the  public  benefits  required  in  the 
general  criteria,  all  industry-relevant  programs  and 
projects  must  identify  and  use  uniform  benefit 
indicators  (including  benefit-cost  ratios)  to  enable 
comparisons  of  expected  benefits  across  programs  and 
projects.  0MB  will  work  with  agencies  to  identify 
these  indicators. 

B.  Programs  and  projects  must  justify  the  appropriateness 
of  federal  investment,  including  the  manner  in  which 
the  market  fails  to  motivate  private  sector 

investment . 

A  lack  of  market  incentives  discourages  private  firms 
from  investing  in  research  where  the  benefits  may 
occur  far  in  the  future,  the  risks  may  be  too  great 
for  non-federal  participants,  or  the  benefits  accrue 
to  the  public  rather  than  private  investors.  Programs 
and  projects  must  demonstrate  that  industry  investment 
is  sub-optimal  and  explain  in  what  way  the  market 
fails  that  prevents  the  private  sector  from  capturing 
the  benefits  of  developing  the  good  or  service. 


C.  Programs  and  projects  must  demonstrate  that  investment 
in  R&D  and  demonstration  activities  is  the  best  means 
to  support  the  federal  policy  goals,  compared  to  other 
policy  alternatives. 

When  the  federal  government  chooses  to  intervene  to 
address  market  failures,  there  may  be  many  policy 
alternatives  to  address  those  failures.  Among  the 
other  tools  available  to  the  government  are 

legislation,  tax  policy,  regulatory  and  enforcement 
efforts,  and  an  integrated  combination  of  these 
approaches.  In  this  context,  projects  to  address 
issues  of  genuine  federal  concern  should  be  able  to 

illustrate  how  R&D  and  demonstration  activities  are 
superior  to  other  policy  tools  in  addressing  federal 

goals,  either  by  themselves  or  as  part  of  an 

integrated  package. 


D.  Programs  and  projects  must  document  industry  or  market 
relevance,  including  readiness  of  the  market  to  adopt 
technologies  or  other  outputs. 
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Programs  must  assess  the  likelihood  that  the  target 
industry  will  be  able  to  adopt  the  technology  or  other 
program  outputs.  The  level  of  industry  cost  sharing  is 
one  indicator  of  industry  relevance.  Before  projects 
move  into  demonstration  or  deployment  stages,  an 
economic  analysis  of  the  public  and  private  returns  on 
the  public  investment  must  be  provided. 


E.  Program  performance  plans  and  reports  must  include 
"off  ramps"  and  transition  points. 

In  addition  to  the  schedules  and  decision  points 
defined  in  the  general  criteria,  program  plans  should 
also  identify  whether,  when,  and  how  aspects  of  the 
program  may  be  shifted  to  the  private  sector. 
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APPENDIX  C.  SCORECARD  CORE  CRITERIA 


(Source :  http : / / www . j  ohn-mercer . com/ pma-stds5 . htm) 

Criteria  for  "Green" 

In  order  to  be  recognized  as  achieving  full  compliance 

with  the  Budget  and  Performance  Integration  initiative  of 

the  President's  Management  Agenda,  the  agency  must  satisfy 

0MB  that  it  meets  each  of  the  following  six  Core  Criteria: 

o  "Senior  agency  managers  meet  at  least  quarterly 
to  examine  reports  that  integrate  financial  and 
performance  information  that  covers  all  major 
responsibilities  of  the  Department.  This 
information  is  used  to  make  decisions  regarding 
the  management  of  agency  programs;" 
o  "Strategic  plans  contain  a  limited  number  of 
outcome-oriented  goals  and  objectives.  Annual 
budget  and  performance  documents  incorporate 
all  measures  identified  in  the  PART  and  focus 
on  the  information  used  in  the  senior 

management  report  described  in  the  first 
criterion; " 

o  "Performance  appraisal  plans  for  at  least  60% 
of  agency  positions  link  to  agency  mission, 
goals  and  outcomes,  effectively  differentiate 
between  various  levels  of  performance,  and 
provide  consequences  based  on  performance;" 
o  "Reports  the  full  cost  of  achieving  performance 
goals  accurately  (+/-10%)  in  budget  and 

performance  documents  and  can  accurately 
estimate  the  marginal  cost  (+/-10%)  of  changing 
performance  goals;"  "Has  at  least  one 
efficiency  measure  for  all  programs;  and" 
o  "Uses  PART  evaluations  to  direct  program 
improvements  and  PART  ratings  are  used 
consistently  to  justify  funding  requests, 
management  actions,  and  legislative  proposals. 

Less  than  10%  of  agency  programs  receive  a 
Results  Not  Demonstrated  rating  for  more  than 
two  years  in  a  row." 
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Criteria  for  "Yellow" 


A  rating  of  "yellow"  is  awarded  when  the  agency  meets 
the  following  criteria: 

o  "Senior  agency  managers  meet  at  least  quarterly 
to  examine  reports  that  integrate  financial  and 
performance  information  that  covers  some  of  the 
major  responsibilities  of  the  Department.  The 
reports  are  used  to  make  decisions  regarding 
the  management  of  Agency  programs;" 
o  "Strategic  plans  contain  a  limited  number  of 
outcome-oriented  goals  and  objectives.  Annual 
budget  and  performance  documents  incorporate 
all  measures  identified  in  the  PART  process;" 
o  "Performance  appraisal  plans  for  SES  and 
managers  link  to  agency  mission,  goals  and 
outcomes,  effectively  differentiate  between 
various  levels  of  performance,  and  provide 

consequences  based  on  performance;" 
o  "The  full  cost  of  achieving  performance  goals 
is  accurately  (+/-10%)  reported  in  budget  and 
performance  documents;" 

o  "At  least  50%  of  agency  programs  rated  by  the 
PART  have  at  least  one  efficiency  measure;" 
o  "PART  ratings  are  used  to  justify  funding 

requests,  management  actions,  and  legislative 
proposals.  No  more  than  50%  of  agency  programs 
receive  a  Results  Not  Demonstrated  rating  for 
more  than  two  years  in  a  row." 

Negative  Conditions  for  "Red" 

A  rating  of  "red"  is  triggered  if  any  one  of  the 

following  negative  Conditions  is  found: 

o  "Senior  agency  managers  do  not  have  a  regular 
process  for  considering  financial  and 
performance  information  when  making  decisions 
regarding  the  management  of  Agency  programs;" 
o  "Strategic  plans  contain  too  many  goals  and 

objectives  to  provide  a  clear  and  focused 
statement  of  Agency  priorities.  Performance 
measures  included  in  annual  budget  and 
performance  documents  do  not  meet  the  standards 
of  the  PART;" 
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o 


"Performance  appraisal  plans  do  not  link  to 
agency  mission,  goals  and  outcomes,  effectively 
differentiate  between  various  levels  of 
performance,  or  provide  consequences  based  on 
performance ; " 
o  "Does  not  have  a  systematic  way  to  estimate  the 
full  cost  of  achieving  performance  goals 
reported  in  budget  and  performance  documents;" 
o  "Less  than  50%  of  agency  programs  rated  by  the 
PART  have  at  least  one  efficiency  measure;"  OR 
o  "Agency  does  not  consistently  use  PART  ratings 
to  justify  funding  requests,  management 
actions,  and  legislative  proposals.  More  than 
50%  of  agency  programs  receive  a  Results  Not 
Demonstrated  rating  for  more  than  two  years  in 
a  row . " 
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